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FLEXIBLE  METHODS  FOR  FUTURE  FORCE  CONCEPT  DEVELOPMENT 

EXECUTIVE  SUMMARY 


Research  Requirement: 

The  U.S.  Army  must  find  methods  to  rapidly  incorporate  technology,  must  make  the 
necessary  systemic  changes  so  that  future  Soldiers  can  use  the  assets  efficiently  and  effectively, 
and  must  do  so  at  a  rate  that  outpaces  our  competitors.  In  short,  technology  must  be  introduced 
to  enhance  rather  than  impair  operational  effectiveness. 

A  key  to  success  is  a  solid  process  for  concept  exploration.  There  is  a  tremendous  need 
for  a  powerful,  yet  flexible,  approach  to  concept  development  and  knowledge  exploration  that 
can  cope  with  the  unique  difficulties  of  envisioning  and  studying  future  concepts.  The  Army 
needs  a  means  to  generate,  elaborate,  refine,  describe,  test,  and  validate  new  concepts  relating  to 
doctrine,  tactics,  techniques,  procedures,  unit  and  team  organization,  job  allocation,  training, 
leader  development,  and  other  aspects  of  technology  integration. 

Procedure: 

Two  tools  were  developed  to  support  elicitation  of  future  tactics.  A  scaled-world  tool  and 
a  concept-development  tool  were  designed  and  developed  to  examine  situations  related  to  the  See 
function  as  it  might  operate  in  the  future.  The  tools  provide  a  means  to  present  individuals  with 
situations  that  stimulate  their  thinking  in  the  context  of  realistic  situations  or  to  present  situations 
in  which  only  a  limited  number  of  variables  are  operative  in  order  to  assess  how  individuals  will 
perform.  Six  scaled-world  events  and  10  concept-development  sessions  were  produced.  The 
scaled-world  events  and  concept-development  sessions  were  formatively  evaluated  using  reviews 
by  military  personnel. 

Findings: 

Based  on  formative  evaluation  (FE)  results,  it  appears  that  both  tools  have  value  for 
concept  identification  and  concept  development.  Feedback  on  both  components  was  very 
positive  and  generally  met  the  project  objectives.  One  exception  is  that  the  components  are  not 
as  easily  modifiable  as  originally  anticipated. 

Utilization  of  Findings: 

The  results  of  this  project  were  transitioned  to  the  Unit  of  Action  Maneuver  Battle  Lab 
(UAMBL).  The  tools  benefit  those  involved  in  the  identification,  refinement,  and  testing  of 
concepts  related  to  the  Army’s  transformation  to  the  Future  Forces  environment.  Both  tools 
serve  an  important  function  in  the  initial  stages  of  Future  Forces  development  by  providing 
mechanisms  that  can  be  employed  in  relatively  simple  settings  using  existing  equipment  in  order 
to  identify  concepts. 
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FLEXIBLE  METHODS  FOR  FUTURE  FORCE  CONCEPT  DEVELOPMENT 


Introduction 

The  U.S.  Army  is  in  the  process  of  transforming  to  a  Future  Force  following  the 
guidelines  laid  out  several  years  ago  by  the  Joint  Chiefs  of  Staff  (JCS1)  in  the  Joint  Vision  2020 
(JCS,  2000).  The  transformation  is  based  on  the  overarching  concept  that  U.S.  forces,  acting 
jointly  and  in  combination  with  forces  from  other  nations,  will  have  the  capability  to  find  and 
defeat  any  enemy  before  that  enemy  poses  a  direct  threat.  A  key  component  of  this  capability 
involves  the  concept  of  information  dominance  which  will  depend,  in  large  part,  on  technological 
advances  in  the  areas  of  advanced  sensors,  artificial  intelligence,  and  network-centric  command 
and  control  systems.  However,  it  will  be  equally  dependent  on  the  ability  of  future  Soldiers  to 
use  these  new  assets  efficiently  and  effectively. 

The  transformation  will  most  likely  be  dramatic,  producing  significant  change  in  how  the 
U.S.  Army  operates;  it  will  be  widespread,  involving  multiple  interdependent  effects  across  the 
range  of  Doctrine,  Organization,  Training,  Materiel,  Leadership  and  Education,  Personnel,  and 
Facilities  (DOTMLPF);  and  it  will  be  continual,  taking  advantage  of  technology  that  will  itself  be 
advancing  rapidly  throughout  the  century.  In  addition,  the  future  of  U.S.  Army  dominance  will 
not  be  as  dependent  as  it  previously  was  upon  the  Department  of  Defense  (DOD)  designing, 
building,  and  protecting  unique  technologies.  Instead,  it  will  most  likely  be  built  on  designing, 
implementing,  and  protecting  the  world's  best  technology  integration  process  (Carter  &  White, 
2000).  This  is  a  different  way  of  doing  business.  It  is  predicated  on  the  notion  that  we  as  a 
nation  will  not  be  able  to  control  the  development  or  proliferation  of  new  technologies  in  a  global 
marketplace.  Rather,  we  will  need  to  become  the  best  and  fastest  integrator  of  new  technologies 
-  into  the  hardware,  into  the  software,  into  our  doctrinal  and  organizational  systems,  and  into  our 
training  and  leader  development  processes. 

A  key  to  success  in  the  Army’s  transformation  is  a  solid  process  for  concept 
development.  The  Army  needs  a  means  to  generate,  elaborate,  refine,  describe,  test,  and  validate 
new  concepts  relating  to  doctrine,  tactics,  techniques,  procedures,  unit  and  team  organization,  job 
allocation,  training,  leader  development,  and  other  aspects  of  technology  integration.  How  do  we 
best  use  the  total  Army  team  -  warfighter,  scientist,  theorist,  analyst,  and  engineer  -  in  the 
development  process?  Generally  the  Army  has  used  two  methods.  The  first  is  to  have  a  strong 
theorist  or  analyst  develop  and  present  new  concepts,  that  is,  an  Army  thinker  thinks  and  then 
writes.  A  good  example  of  this  is  provided  by  Wass  de  Czege  and  Sinnreich  (2002).  While 
strong  creative  thinkers  are  clearly  a  vital  part  of  the  effort,  some  drawbacks  are  evident  in  the 
“just  let  thinkers  think”  method  of  concept  development.  The  results  are  often  very  general,  have 
not  benefited  from  concrete  attempts  to  implement  the  ideas,  have  not  uncovered  difficulties 
through  implementation,  can  be  hard  to  communicate,  and  do  not  easily  benefit  a  team  of 
specialized  contributors  working  jointly.  Several  efforts  less  successful  than  the  above-cited 
example  could  also  be  mentioned,  resulting  in  “cut-and-paste”  concept  documents  that  were 
poorly  thought  out,  too  general,  impractical,  unrealistic,  fragmented,  disjointed  and  contradictory. 


1  A  list  of  all  acronyms  used  in  this  report  is  included  in  Appendix  A. 
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A  second  Army  method  is  to  stand  up  a  replica  of  the  new  system  and  conduct  a  unit 
exercise  in  simulation,  such  as  a  focused  Combat  Training  Center  (CTC)  rotation  or  Advanced 
Warfighter  Experiment  (AWE).  Such  full-scale  events  can  be  effective  demonstrations  of 
moderate-  to  well-developed  concepts  but  have  disadvantages,  especially  in  the  earlier  stages  of 
concept  formation.  They  are  expensive  and  require  great  effort  and  coordination  to  conduct,  do 
not  allow  flexible  manipulation  of  variables,  present  great  measurement  difficulties,  and  lack  the 
repetition  necessary  to  reach  well-founded  conclusions.  Further,  the  complexity  of  conducting 
the  event  tends  to  overwhelm  the  experimental  intent. 

There  is  a  need  to  investigate  concept  development  related  to  using  methods  that  provide 
valid  and  reliable  information  while  avoiding  the  difficulties  described  above.  One  approach  is 
to  construct  an  environment  that  transforms  the  scale  in  which  situations  can  be  presented 
realistically.  Lickteig  et  al.  (2002)  make  a  strong  case  for  developing  such  an  environment. 

They  describe  small-scale  transformation  environments  as  “empirical  venues  that  afford  users, 
researchers,  and  developers  the  ability  to  customize  tasks  and  conditions  in  order  to  iteratively 
explore  and  transform  concepts  into  viable,  human-centric  solutions”  (p.  6).  They  further  state 

such  an  environment  can  serve  as  a  middle  ground  between  the  situational 
complexity  inherent  in  field  research  that  resists  definite  conclusions,  and  the 
situational  paucity  of  laboratory  research  that  defies  useful  conclusions  (Ehret, 

Gray,  &  Kirschenbaum,  2000).  By  design,  it  preserves  key  functional 
relationships  based  on  questions  of  interest  to  evaluators  or  trainers,  while  paring 
away  other  functions  that  might  confound  answering  those  questions  (p.  8). 

A  scaled  environment,  according  to  Lickteig  et  al.  (2002),  serves  at  least  two  important 
functions.  It  provides  a  breeding  ground  in  which  to  identify  and  refine  innovations  for  larger 
environments,  and  it  provides  a  proving  ground  in  which  to  assess  and  resolve  key  issues  from 
larger  environments. 

The  idea  of  the  scaled  environment  serving  as  a  breeding  ground  is  consistent  with  the 
discussion  included  in  the  Joint  Operations  Concept  -  Full  Spectrum  Dominance  Through  Joint 
Integration  (2003),  which  focuses  on  identification  of  enabling  concepts  or  subordinate  concepts 
that  will  need  to  be  developed  with  sufficient  detail  to  directly  link  to  capabilities.  Identification 
of  such  enabling  concepts  is  a  primary  function  of  scaled  transformation  environments.  Gold 
(1999)  further  supports  this  use  of  scaled  transformation  environments  and  warns  against 
focusing  on  so-called  “Superbowl”  culminating  events.  Instead,  the  concept  identification 
process  should  be  built  on  a  “broad-based  iterative  approach  using  a  variety  of  tools,  simulations, 
and  venues  that  can  lead  to  early  discovery.  Early  on,  searching  for  questions  is  more  important 
than  finding  answers”  (p.  26). 


Purpose  and  Objectives 

These  thoughts  and  arguments  about  concept  exploration,  identification,  and  development 
come  at  a  critical  time  for  the  U.S.  Army  as  it  engages  in  a  major  effort  to  transform  its 
operations.  The  U.  S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI),  as 
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part  of  its  Science  and  Technology  Objective  on  Methods  and  Measures  of  Commander-Centric 
Training,  conducted  an  analysis  to  describe  the  C4ISR  functional  and  task  requirements  for  FCS 
(Lickteig,  Sanders,  Durlach,  Lussier,  &  Carnahan,  2003).  Some  examples  of  the  Command, 
Control,  Communications,  Computers,  Intelligence,  Surveillance  and  Reconnaissance  (C4ISR) 
functions  are:  control  and  interpret  input  from  a  heterogeneous  set  of  advanced  sensors  to 
mentally  construct  an  accurate  picture  of  enemy  activity  and  intent  {See),  control  the  movement 
and  activity  of  friendly  manned  and  unmanned  systems  to  maintain  desired  movement  rates  and 
formations  {Move),  distribute  a  variety  of  indirect  and  direct  effects  over  a  set  of  targets  {Fire), 
assess  the  effectiveness  of  a  plan  and  the  combat  instruction  sets  provided  to  robotic  elements 
{Rehearse),  and  other  functions  such  as  Sustain,  Train,  Communicate,  Adapt,  and  Plan.  A  set  of 
tools  is  needed  that  can  be  used  to  explore  selected  command  group  functional  performance 
issues  in  a  methodical  fashion.  The  tools  must  be  flexible,  that  is,  they  must  be  capable  of  being 
readily  tailored  to  respond  to  a  wide  range  of  conceptual  constructions.  The  current  project 
developed  two  tools  -  scaled  world  events  and  the  concept  development  sessions.  Scaled  world 
events  use  simulation  to  perform  a  function  in  isolation.  Concept  development  sessions  use 
realistic  vignettes  without  simulation  to  allow  a  participant  to  generate  and  evaluate  ideas  about 
future  tactical  issues. 


The  See  Function 

The  tools  were  developed  to  address  the  “See”  function.  The  application  of  emerging 
sensor  capabilities  will  greatly  affect  how  Army  personnel  see  the  battlefield  in  the  near  future. 
New  technology  should  enhance  collective,  real-time  common  understanding  of  the  battlefield 
situation  over  great  distances  and  allow  commanders  to  make  rapid  and  accurate  maneuvers  in 
any  operating  environment.  The  enhanced  capabilities  will  present  information  from  robotic  air 
and  ground  sensors,  and  from  humans,  using  graphics  and  text.  The  deluge  of  information, 
however,  will  require  users  to  ‘fuse’  information  from  these  various  sources  into  an 
understanding  of  the  situation.  This  alone  requires  innovative  methods  of  training,  new  concepts 
of  presentation  and  succinct  approaches  to  information  presentation.  Additionally,  all  other 
battlefield  functions  rely  on  input  from  the  See  function  for  successful  execution;  thus,  concepts 
and  methods  developed  relating  to  the  See  function  will  have  broad  effect  and  application  within 
Future  Forces. 

An  initial  statement  of  the  See  function  was  formulated  as  follows: 

The  Command  group  controls  and  interprets  information  to  mentally  construct  an 
accurate  picture  of  the  environment  and  of  threat  dispositions,  activities, 
capabilities,  and  intent  in  relation  to  friendly  dispositions,  activities,  capabilities, 
and  desires. 

The  function  was  analyzed  into  its  component  tasks  and  activities  as  projected  for  the 
environment,  and  the  tasks  and  activities  were  then  examined  to  identify  those  that  would  be 
included  in  a  synthetic  environment  for  concept  exploration.  Some  components  of  the  See 
function  that  were  identified  are:  decide  what  to  look  for,  decide  where  to  look,  decide  what  to 
look  with,  set  filters,  fuse  information  on  screen(s),  review  information,  interpret  information 
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(analyze  and  synthesize),  evaluate  information,  track  forces,  verily  information,  share 
information,  and  identify  information  gaps. 

Scaled-World  Tool 

The  scaled-world  tool  allows  participants  to  deploy  and  control  various  robotic  sensors  to 
perform  surveillance  and  reconnaissance  tasks  in  isolation,  that  is,  separate  from  a  full  tactical 
context.  Various  types  of  robotic  entities  (aerial,  mobile  ground,  or  immobile  ground),  equipped 
with  different  types  of  sensors  (e.g.,  visual,  infrared,  auditory),  are  represented.  It  supports  the 
conduct  of  scaled-world  events  (SWE).  Scaled-world  events  are  preset  mission  scenarios  in 
which  an  individual  or  small  team  performs  a  task  or  set  of  tasks  that  emulate  and  isolate  selected 
components  of  the  See  function  in  a  laboratory  setting.  Each  scenario  specifies  the  mission  goal 
and  the  type  of  sensors  that  are  available.  For  example,  a  SWE  might  require  the  participants  to 
locate  a  camouflaged  enemy  unit.  The  participant  can  see  the  location  of  their  tasked  sensors  on 
an  electronic  map.  Simultaneously,  in  additional  windows,  they  can  view  the  sensor  images  and 
associated  data.  The  primary  purpose  of  the  scaled-world  tool  is  to  present  an  accurate  portrayal 
of  the  task  requirements  so  that  concept  development  is  grounded  by  a  realistic  appreciation  for 
the  human  performance  difficulties  associated  with  conducting  the  task.  The  scaled-world  tool 
also  allows  researchers,  by  measuring  and  assessing  the  performance  of  participants,  to  draw 
conclusions  that  are  applicable  to  future  Army  conditions.  Issues  such  as  human  performance 
capabilities,  training  requirements,  and  task  requirements  can  be  investigated. 

Concept-Development  Tool 

The  concept-development  tool  supports  the  conduct  of  concept-development  sessions 
(CDS).  The  notion  of  a  CDS  is  that  of  a  snapshot,  or  a  series  of  snapshots,  or  a  short 
presentation  that  places  a  decision-making,  problem  solving,  or  evaluation  requirement  on  the 
participants.  A  CDS  does  not  respond  dynamically  to  participant  input.  However,  CDSs  can 
portray  highly  realistic  situations  and  entities  that  are  often  very  difficult  to  represent  well  in 
tactical  simulation,  for  example,  underground  caves,  car  bombs,  cultural  factors,  and  situations 
arising  from  support  and  stability  operations. 

The  CDSs  and  the  concept-development  tool,  which  are  independent  of  the  scaled-world 
tool,  place  the  participant(s)  in  a  situation  in  which  a  decision,  evaluation,  or  other  problem  is 
posed  that  relates  to  the  target  functions  and  tasks  represented  in  the  scaled-world  environment. 
Each  CDS  includes  a  specification  or  questionnaire  for  collecting  participant  thoughts,  questions, 
conclusions,  and  recommendations. 

The  CDSs  can  be  used  in  a  variety  of  ways,  including  the  following  broad  areas: 

•  To  represent  “What  now?”  instances,  portraying  a  military  situation  and  asking  the 
participant(s)  to  generate  and  evaluate  courses  of  action  (COA). 

•  As  “What  if?”  situations  that  explore  the  range  of  employment.  For  example,  if  the 
enemy  used  x  method  of  infiltrating,  how  should  we  adjust  surveillance  tactics?  How 
would  the  sensor  plan  need  to  be  changed  under  .g  terrain  and  weather  conditions? 
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•  As  acquisition-oriented  human  factors  tools.  For  example,  if  the  Army  were  to 
develop  z  sensor  capability,  how  should  that  be  employed? 

•  To  assess  the  effect  of  infusions  of  new  technologies  by  presenting  realistic  situations 
and  obtaining  participant  reactions. 

Comparison  of  Scaled-World  and  Concept-Development  Tools 

The  scaled-world  and  concept-development  tools  complement  each  other  in  their 
strengths  and  weaknesses.  An  SWE  represents  a  “scaled-down”  version  of  a  situation  in  which 
variables  of  interest  are  isolated  and  complicating  variables  are  eliminated  to  the  extent  possible. 
In  the  SWE,  the  participant  is  able  to  manipulate  sensors  and  perform  the  task  in  an  authentic 
manner,  allowing  the  participant  to  directly  experience  human  performance  requirements.  A 
CDS  presents  a  complex  situation  with  a  full  range  of  tactical  considerations.  The  participant  is 
able  to  explore  realistic  situations  but  cannot  directly  interact  with  them.  The  CDS  facilitates  an 
environment  in  which  new  concepts  can  be  explored,  without  being  constrained  by  simulation 
implementation.  It  allows  the  researcher  to  define  and  focus  emerging  future  issues  and  to 
accumulate  the  judgments  of  participants.  Experienced  together,  concept  development  is 
enhanced  as  the  case-based  quality  of  the  CDS  stimulates  exploration  of  ideas  and  the 
performance  requirements  of  the  SWE  impose  realistic  evaluation  on  the  feasibility  of  the  ideas. 

Scaled-World  Events 

Six  SWEs  were  produced  based  on  capabilities  expected  to  exist  in  the  future.  Each 
SWE  included  a  narrative  and  graphical  presentation  of  the  situation  involved,  the  anticipated 
activities  of  the  participants,  and  anticipated  performance  measures.  For  example,  factors  that 
are  expected  to  affect  performance  in  the  scaled-world  include  threat  type  and  size,  number  and 
types  of  sensors  available,  and  rate  of  information  being  presented  to  participants. 

The  first  SWE  is  designed  at  the  level  of  a  Brigade  Combat  Team  (BCT)2.  This  event  is 
very  basic  and  serves  the  function  of  orienting  participants  to  the  scaled- world  environment.  The 
only  variables  that  are  manipulated  are  the  number  and  types  of  sensors  available.  Participants 
begin  the  event  with  a  terrain  map  showing  their  area  of  operations  (AO).  They  receive  a 
message  from  higher  informing  them  that  threat  force  movement  in  an  adjacent  AO  was  detected 
with  the  threat  possibly  moving  into  their  AO.  Participants  attempt  to  locate  and  identify  the 
threat  forces  by  deploying  sensors  and  obtaining  information  from  sensors  owned  by  higher  or 
adjacent  units.  The  threat  forces  consist  of  a  stationary  and  camouflaged  scout  platoon.  The 
event  is  conducted  during  daylight  conditions.  Skies  are  clear  with  no  obstacles  to  visibility 
related  to  weather.  The  event  concludes  when  the  unit  detects,  locates,  and  identifies  the  threat 
force.  A  brief  description  of  each  SWE  is  shown  in  Table  1 .  Detailed  descriptions  of  all  six 
SWEs  are  in  Appendix  B. 


2  In  the  original  materials,  the  BCT  is  referred  to  as  Unit  of  Action  (UA). 
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Table  1.  Description  of  SWE  tasks 


SWE  1:  Detect  stationary  troops. _ 

SWE  2:  Locate  mobile-based  biological  threat. _ 

SWE  3:  Determine  enemy  course  of  action. _ 

SWE  4:  Determine  whether  shots  fired  are  a  party  or  a  battle. _ 

SWE  5:  Identify  when  a  critical  event  occurs. _ 

SWE  6:  Conduct  battle  damage  assessment. _ _ 

Representative  screens  from  the  scaled- world  tool  are  shown  in  Figures  1  and  2.  The 
scaled- world  tool  user  interface  is  composed  of  four  panels,  as  shown  in  Figure  1.  The  first  is  the 
mode  selection  area  panel,  which  displays  in  the  upper  left  portion  of  the  screen  and  provides  the 
user  with  the  capability  to  manipulate  the  map,  deploy  sensors,  and  select  sensor  feeds  to  view. 
Those  functions  are  controlled  by  the  “Map  Navigation,”  “New  Sensors,”  and  “Current  Sensors” 
buttons  at  the  top  of  the  screen.  For  aerial  and  mobile  ground  sensors,  users  can  establish  a  route 
of  travel  as  well  as  a  loiter  pattern  that  the  sensor  will  follow.  The  second  panel,  on  the  right,  is 
the  map  area  panel,  which  takes  up  most  of  the  screen.  The  map  represents  a  portion  of  Fort 
Hood  terrain.  Following  sensor  deployment,  the  map  displays  the  sensor’s  location  as  well  as  its 
type.  For  example,  in  the  center  of  the  map  area  panel  of  Figure  1,  three  deployed  sensors  are 
shown.  The  third  panel,  in  the  lower  left  is  referred  to  as  the  situation  area ,  and  contains  a  text 
description  of  the  SWE  situation.  The  fourth  panel,  lower  right,  is  the  sensor  output  area  that 
displays  messages  sent  by  non-visual  sensors  (e.g.,  seismic,  audio,  biological  agent).  A  message 
is  displayed  in  the  sensor  output  panel  whenever  a  sensor  comes  within  range  of  a  target  it  is 
capable  of  detecting.  The  user  can  then  display  the  sensor  output  for  additional  information 
including  playback  of  audio  recordings  made  by  the  sensor.  For  each  visual  sensor  deployed,  a 
new  window  displays  the  corresponding  sensor  image  as  shown  in  Figure  2.  Three  types  of 
visual  images  are  available:  a  daylight  visual  image,  an  infrared  image,  and  a  thermal  image.  A 
set  of  instructions  (See  Appendix  C)  provides  information  on  loading  exercises,  navigating  the 
map,  and  deploying  and  monitoring  sensors. 
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Figure  1 .  Scaled- world  tool  interface  showing  the  user  panels. 


The  scaled-world  tool  can  collect  limited  data  in  a  Microsoft  Access®  database.  It  can 
record  the  sensors  that  were  deployed  and  whether  a  sensor  ever  came  within  range  to  detect  the 
target.  It  cannot,  however,  record  whether  the  user  actually  detected  the  target. 
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Figure  2.  Scaled- world  tool  with  sensor  feed  displayed. 

The  scaled- world  tool  and  events  were  developed  using  Java  and  open-source  software 
including: 

•  Gentoo  Linux  (Operating  System). 

•  Netbeans  3.4  -  3.5  (Integrated  Development  Environment). 

•  OpenMap  (Map  Application  Programming  Interface). 

•  IzPack  (Installer). 

•  Sun  JDK  1 .4.1-1 .4.2  (Java  Virtual  Machine). 

•  OpenOffice  (Office  Suite) 

In  addition,  terrain  and  visual  imagery  maps  for  Fort  Hood  were  obtained  from  the  National 
Imagery  and  Mapping  Agency  (NIMA). 

Developing  new  events  or  modifying  existing  ones  requires  some  level  of  expertise  with 
Java.  Simple  modifications  such  as  changing  the  characteristics  of  a  sensor  or  moving  the 
location  of  a  target  involve  relatively  minor  changes  to  lines  of  existing  code.  Adding  new 
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sensors  or  targets,  making  major  modifications  to  an  event,  or  creating  a  new  event 
requires  more  extensive  coding. 

Concept-Development  Sessions 


A  concept  development  session  (CDS)  is  a  facilitated  discussion  revolving  around  a 
question  or  issue.  Each  CDS  presents  a  depiction  of  a  situation  designed  to  start  the  participants 
thinking  about  how  the  Army  might  function  in  the  future.  The  goal  is  to  generate  concepts  for 
future  operations  that  can  be  used  to  develop  ideas  about  tactics,  techniques,  and  procedures 
(TTP),  e.g.,  how  a  particular  capability  might  best  be  used.  Each  participant  is  shown  a  problem 
scenario  relevant  to  an  important  Army  issue  and  asked  to  think  through  the  specific  situation. 
Then,  a  broader  discussion  is  initiated  to  include  considerations  beyond  the  scenario  presented  in 
order  to  test  the  generality  of  the  solutions  devised  for  the  specific  scenario.  A  succession  of 
participants  confronting  the  same  CDS  helps  to  refine  the  concepts.  In  total,  ten  CDSs  were 
developed. 


Each  session  starts  with  an  instruction  screen  that  identifies  the  role  the  participant  is  to 
play,  the  tactical  mission,  and  the  goal  of  the  session.  Figure  3  shows  the  instruction  screen  for 
one  of  the  CDSs. 


Instructions 


Vour  role  and  mission  In  the  session; 

You  will  be  ploying  the  role  of  the  commander  of  C  Troop  (Trp),  1st  Combined  Arms  (CA)  BN. 
2nd  Unit  of  Action  (UA,  Brigade-like),  4th  Unit  of  Employment  (UE,  Division-like).  Your  unifs 
tactical  task  Is  to  determine  the  nature  of  the  threat  In  Uphlan  and  report  to  the  battalion. 


The  goal  of  this  session:  '  ■  fa 

;! 

The  goal  of  this  session  Is  to  stimulate  your  thoughts  and  to  provide  Information  that  will  help 
shape  the  way  future  forces  will  operate  to  complete  their  mission.  More  specifically ,  the  goal  Is 
to  come  up  with  Ideas  for  determining  the  kind  of  information  you  would  want  to  know  for  the 
scenario  described,  and  the  kind  of  assets  you  would  want  for  obtaining  that  Information.  As  j 
you  think  through  this  session,  your  thoughts  are  the  primary  data  It  Is  designed  to  capture.  j 

You  are  strongly  encouraged  to  " think  out  loud ‘  and  to  think  "future  forces. "  We  are 
particularly  Interested  In  your  Inputs  related  to  the " See  the  Battlefield ”  function,  however,  feel  ;:j 
free  to  explore  other  areas.  Ask  questions  of  the  observers  at  any  time. 

When  you  have  finished  examining  the  information,  inform  the  observers.  They  will  ask  you 
questions  that  will  help  you  verbalize  your  thoughts.  There  are  no  Incorrect  or  bad  answers;  jf  I 

all  of  the  Information  you  provide  will  be  valuable  in  helping  to  define  how  future  forces  will 
operate.  ' 

When  you  are  ready  to  start  the  session,  click  Begin. 


Figure  3.  Typical  instruction  screen  from  a  concept-development  session. 
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When  the  participants  click  the  Begin  link,  they  see  additional  screens  such  as  those 
shown  in  Figures  4,  5  and  6.  The  CDSs  are  designed  to  run  in  a  dual-screen  environment.  “Hot 
links”  on  each  screen  allow  the  participant  to  obtain  additional  information.  The  participants  are 
given  as  much  time  as  desired  to  examine  the  information  and  are  encouraged  to  “think  out  loud” 
as  they  study  the  CDS  scenario.  After  reading  the  introductory  page  on  the  computer  screen,  the 
participants  work  through  the  CDS  by  studying  and  analyzing  the  information  provided, 
formulating  questions  and  plans,  and  indicating  the  need  for  additional  information. 

The  first  CDS,  for  example,  presents  a  tactical  scenario  that  includes  a  detailed  written 
account  of  civilian  activities  in  a  town  known  to  harbor  threat  personnel  on  existing  blacklists. 
The  participants  view  a  series  of  photographs  including  groups  of  armed  persons  congregated 
near  the  town’s  central  hall.  They  have  access  to  maps  along  with  graphic  displays  showing  their 
unit  task  organization  and  additional  assets  available.  They  are  told  that  the  unit’s  task  in  this 
situation  is  to  determine  the  nature  of  the  threat  in  the  town  and  make  a  report  to  the  Battalion 
Commander. 

The  structured  interview  elicits  responses  aimed  at  shaping  concepts  and  tactics, 
techniques,  and  procedures  related  to  the  situation.  The  specification  of  each  CDS  contains  a 
number  of  pre-designed  questions  for  the  facilitator  to  ask.  However,  in  each  session  facilitators 
are  free  to  ask  additional  clarifying  questions  and  follow-up  on  the  course  of  participants’  unique 
ideas.  Some  pre-designed  questions  used  for  the  first  CDS  are: 

1 .  What  steps  would  you  take  to  complete  this  task?  (Follow-up  and  clarify  as  necessary) 

2.  What  additional  assets/capabilities,  if  any,  would  you  liked  to  have  had  to  complete  this  task, 
and  how  would  you  have  employed  those  assets? 

3.  Is  there  information  that  you  did  not  have  that  would  have  helped  you  complete  this  task? 

What  information?  How  could  it  best  be  presented? 

4.  Is  this  an  appropriate  task  for  the  unit  indicated?  Should  it  be  a  higher  level  task?  Lower 
level? 

Appendix  D  contains  descriptions  of  all  10  CDSs. 
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Concept  Development  Session  1 


Retvmto  IreWsfcre 


General  Situation 

There  has  been  general  unrest  in  Rediand  for  9 
months,  refugees  are  common  in  all  provinces, 
armed  ethnic  gangs  are  operating  near  key  urban 
areas,  and  Whiteland  (Redland's  main  threat)  have 
conducted  military  maneuvers  near  the  international 
border.  Government  and  religious  leaders  are 
attempting  to  maintain  peace  but  are  fighting  a 
growing  influence  from  factions  against  the 
government. 

The  government  of  Rediand  has  requested  military 
assistance  from  the  United  States  to  help  establish 
stability  in  and  around  the  most  troubled  provinces 
near  the  Rediand  and  Whiteland  border. 

The  terrain  in  the  local  area  is  gently  rolling  with 
extensive  wooded  areas  on  higher  ground.  Weather 
will  continue  to  be  favorable  for  air  and  ground 
operations  for  the  next  46-72  hours  with  low  winds, 
excellent  cross  country  mobility  and  moderate 
temperatures. 

Objectives: 

•  Whiteland  -  intervene  unilaterally  to  stabilize 
the  country  and  gain  control  of  its  resources 
and  people. 

»  Rediand  -  desire  to  maintain  control  of  the 
country,  move  to  a  more  democratic  style  of 
government  and  create  a  close  relationship 
with  the  West. 

•  United  States  -  wish  to  stabilize  the  region, 
keep  Whiteland  from  intervening,  promote 
democracy  in  the  region,  and  support  the 
sitting  Redlands  government. 

Threat: 
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Figure  4.  Typical  information  screen  from  a  concept-development  session  with  hot  links  to  bring 
up  additional  information  about  the  scenario. 


11 


Figure  5.  Typical  supporting  map  from  a  concept-development  session  with  hot  links  at  the 
bottom  allowing  access  to  additional  maps  and  pictures. 


Figure  6.  Aerial  photo  of  the  area  of  operations  from  a  concept-development  session  1  with  hot 
links  at  the  bottom  allowing  access  to  additional  maps  and  pictures. 
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One  concept  for  use  of  a  CDS  is  as  follows.  Initially,  one  person  at  a  time  participates. 
After  several  participants,  responses  and  process  observations  are  used  to  modify  the  CDS  to 
allow  it  to  become  more  realistic,  more  focused,  and  to  incorporate  participants’  ideas  for 
additional  trials.  This  process  of  refinement  and  presentation  is  repeated  until,  in  the  judgment  of 
the  researcher,  no  new  information  is  being  generated.  At  that  point,  final  modifications  are 
made  and  the  CDS  is  presented  to  small  groups.  It  is  anticipated  that  additional  information 
might  be  forthcoming  from  the  group  presentation.  Again,  the  process  is  repeated  until  it  reaches 
the  point  of  diminishing  returns.  After  analysis  of  data  from  multiple  participants,  a  composite 
response  to  the  issue  is  produced.  Further  exploration  may  use  new  CDSs  or  a  SWE  to 
systematically  test  the  group  solution.  To  facilitate  administration  of  CDSs,  an  Administration 
Guide  was  developed  (see  Appendix  E).  This  guide  provides  detailed  information  on  presenting 
sessions,  collecting  data,  modifying  existing  sessions,  and  creating  new  ones.  (See  Shadrick, 
Lussier,  &  Hinkle,  2005)  for  further  discussion  concept  development  in  future  environments.) 

The  concept  development  tool  was  designed  to  allow  for  modular  integration  of  new 
CDSs  in  a  fashion  that  allows  for  easy  substitution  of  new  sessions  and  easy  modification  of 
existing  sessions.  The  CDSs  themselves  were  developed  using  commercial  off-the-shelf 
software  including  Bluesky  ®  RoboHTML  Help™,  Microsoft  ®  PowerPoint™,  and  Adobe  ® 
Photoshop  ™. 

Formative  Evaluation  of  the  Concept  Development  Sessions 

To  evaluate  whether  the  CDS  concept  was  feasible,  seven  trials  were  conducted  with  nine 
participants.  Each  trial  included  two  CDSs  and  was  approximately  two  hours.  The  participants 
included  one  colonel,  two  majors,  five  captains,  and  one  retired  colonel.  Five  sessions  were 
conducted  with  a  single  participant;  two  sessions  were  conducted  with  two  participants.  All  the 
sessions  were  facilitated  by  a  military  subject-matter-expert  and  a  research  psychologist. 

A  number  of  conclusions  may  be  drawn  from  the  formative  evaluation.  Encouraging 
results  were  obtained  on  the  ability  of  the  CDS  format  to  elicit  participant  responses.  Some 
participants  were  stimulated  by  the  CDSs  to  provide  extremely  unique,  creative,  and  well- 
reasoned  responses.  There  were,  however,  considerable  individual  differences.  Some 
participants  -  even  some  who  had  spent  time  studying  the  Army’s  emerging  future  force 
doctrinal  concepts  -  had  difficulty  critically  evaluating  those  concepts.  Some  participants 
appeared  to  be  able  to  transition  into  looking  at  the  situation  from  the  viewpoint  of  how  things 
might  be  in  the  future  (2015-2020)  quite  readily.  Others  required  considerable,  and  even 
forceful,  encouragement.  Soldiers  have  a  tendency  to  look  at  situations  the  way  they  have  been 
trained  (i.e.,  mission,  enemy,  terrain,  troops,  time,  civilians)  and  they  often  tend  to  focus  on  a 
COA  analysis.  In  a  situation  such  as  that  provided  in  the  CDS,  it  may  require  some  training  or 
encouragement  to  lead  their  thinking  toward,  “How  else  could  we  do  things  particularly  if  we  had 
access  to  assets  that  don’t  even  exist  today?”  On  the  other  hand,  once  they  did  start  thinking 
futuristically,  they  were  (for  the  most  part)  very  creative.  They  came  up  with  ideas  for  a  variety 
of  capabilities  including  sensors  that  could  “see  through  things,”  that  could  sense  the  presence  of 
humans  through  exhaled  gasses,  and  that  could  talk  to  people  in  their  own  language.  If  the 
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function  of  the  CDSs  is  to  stimulate  individuals  to  think  creatively  and  futuristically,  they  appear 
to  work. 

Concerning  the  CDS  process  itself,  comments  were  generally  positive  as  to  its  use  as  a 
concept  development  tool.  Most  participants  either  directly  or  indirectly  indicated  that  the  CDS 
tool  would  work.  On  the  other  hand,  specific  comments  regarding  each  CDS  tested  were 
received,  including  some  that  conflicted  (e.g.,  too  much  information;  not  enough  information.) 

Modifications  to  the  Tools 

This  section  discusses  considerations  for  future  modification,  enhancement,  and 
expansion  of  the  SWE  and  the  CDS  tools. 

In  the  case  of  the  SWE,  automated  data  collection  capabilities  are  limited.  The  system 
can  record  when  and  where  sensors  were  placed  and  whether  they  would  ever  have  been  capable 
of  detecting  a  target.  It  does  not  actually  record  whether  a  target  was  detected;  that  would  require 
direct  observation  or  direct  reporting.  A  feature  could  be  added  to  the  SWE  that  would  allow  the 
participants  to  indicate  targets  detected  and  their  locations.  The  system  would  be  able  to  record 
scenario  events  and  certain  aspects  of  human-computer  interaction  in  a  log  file  allowing  the 
researcher  to  review  the  session  in  speeded  up  time  and  to  pause  and  restart  (e.g.,  VCR  like 
controls).  Replay  of  map  screen  and  sensor  information  windows  viewed  by  the  player  would  be 
included.  Time  into  the  mission  should  be  continuously  displayed.  The  playback  should  also 
display  target  information  on  the  map,  annotations  regarding  the  time  of  key  events  such  as  a 
target  falling  inside  sensor  range,  time  and  content  of  operator  reports,  and  whether  these  were 
operator  initiated  or  prompted.  Observers  of  the  playback  should  be  able  to  toggle  sensor  fans 
and  UV  routes  on  or  off.  Screenshots  of  the  playback  should  be  easily  captured  and  saved  for 
later  analysis,  e.g.,  analysis  of  the  amount  of  area  covered  by  sensors  at  a  particular  time. 

A  number  of  suggestions  would  improve  the  sensor  behavior  in  the  SWE.  Currently,  the 
operator  can  set  a  route  for  each  platform.  When  the  route  is  entered,  the  platform  immediately 
starts  on  its  mission  and  the  route  disappears  from  the  map.  The  fact  that  the  route  disappears 
makes  it  difficult  for  the  operator  to  coordinate  the  routes  of  multiple  platforms.  The  operator 
should  be  given  the  ability  to  display  routes  (toggle  on  or  off)  and  to  display  sensor  fans  (toggle 
on  or  off)  to  show  area  of  sensor  coverage.  The  fact  that  the  sensor  deploys  immediately  after  the 
route  is  set  makes  it  difficult  for  the  operator  to  synchronize  multiple  platforms.  A  planning 
mode  would  allow  better  control  of  sensor  missions.  Also,  the  SWE  should  be  upgraded  to  make 
it  easier  to  retask  sensors  mid-mission  (speed,  altitude,  and  route). 

In  the  SWE,  participants  can  control  multiple  sensors  and  view  the  images  from  those 
sensors  in  separate  windows.  It  can  be  difficult  to  associate  the  sensors  on  the  map  with  their 
corresponding  images.  Information  on  the  map  display  should  be  better  linked  to  information  in 
the  individual  sensor  windows,  for  example,  passing  the  mouse  over  a  sensor  window  could 
highlight  the  location  of  the  associated  UV  on  the  map.  Conversely,  clicking  on  a  sensor  on  the 
map  could  cause  some  clear  indication  in  associated  sensor  window,  or  open  the  sensor  window 
if  it  is  not  already  open. 
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Additional  improvements  to  the  SWE  include  adding  a  feature  to  have  the  sensors 
automatically  return  “home”  after  a  set  time  or  after  completing  a  pattern;  incorporating 
additional  loiter  patterns;  and  having  mobile  sensors  begin  their  routes  from  a  point  of  departure 
determined  by  scenario  and  represented  on  the  map  when  the  scenario  is  initiated. 

Further  suggestions  for  SWE  improvement  are  converting  the  latitude/longitude  process 
into  a  grid  coordinate  system  more  familiar  to  Soldiers  and  modifying  the  user  interface  to  supply 
information  on  the  coordinates  for  specific  locations,  e.g.,  a  text  window  could  display  the 
coordinates  of  the  mouse  cursor.  Also,  the  user  interface  should  be  modified  to  indicate  the 
current  zoom  level. 

The  CDS,  because  of  its  inherent  flexibility,  did  not  result  in  such  an  extensive  list  of 
desirable  modifications.  One  goal  for  the  CDSs  was  not  fully  met  -  the  goal  of  making  them 
easily  modifiable.  One  notion  behind  the  approach  was  that  as  users  started  to  identify  concepts 
and  procedures  for  a  CDS,  the  CDS  could  be  easily  modified  to  incorporate  their  ideas.  The 
current  CDSs  are  modifiable,  but  only  to  a  limited  extent.  As  the  computer-based  CDSs  were 
constructed,  the  developers  became  too  tempted  to  add  features  to  enhance  presentation  and  ease 
of  use  and  lost  sight  of  the  goal  to  make  the  CDSs  easily  modifiable.  To  modify  the  CDSs,  a 
good  working  knowledge  of  PowerPoint  and  Compiled  Help  Files  is  needed.  Future  effort  will 
need  to  be  directed  at  how  best  to  achieve  the  goal  of  ready  modifiability,  including  the 
possibility  of  developing  and  presenting  events  and  sessions  in  completely  different  ways. 

Conclusions 

This  report  described  a  project  to  develop  tools  for  generating  and  testing  concepts  and 
ideas  for  situations  or  environments  that  are  beyond  the  scope  of  current  tactics,  techniques,  and 
procedures.  It  examined  a  process  for  exercising  actions  in  a  relatively  controlled  environment  in 
which  only  a  limited  number  of  variables  are  allowed  to  have  impact.  It  also  examined  another 
process  for  presenting  situations  or  scenarios  in  a  relatively  detailed  manner  in  order  to  stimulate 
the  thinking  of  Soldiers  and  others  who  can  provide  valuable  information  related  to  the  current 
transformation  efforts  underway  in  the  U.S.  Army.  Both  the  scaled- world  and  the  concept- 
development  tools  were  formatively  evaluated  and,  based  on  initial  analysis,  appear  to  be  viable 
methods  for  concept  identification,  exploration,  and  testing. 

As  far  as  future  research  directions  are  concerned,  several  possibilities  could  be  pursued. 
Primarily,  the  current  tools  could  benefit  from  additional  testing.  The  effectiveness  and 
efficiency  of  the  approach  has  not  been  empirically  demonstrated.  Studies  that  compare  the  tools 
to  more  conventional  methods  are  important.  If  the  approach  can  be  shown  to  be  more  flexible, 
faster,  or  more  direct  in  providing  reasoned  conceptual  information  related  to  systems, 
operations,  organizations,  and  doctrine,  then  it  will  establish  for  itself  a  position  of  advantage  for 
future  developers  and  decision-makers. 

Testing  should  be  conducted  with  a  variety  of  people.  In  this  initial  trial,  we  found 
considerable  variability  among  the  nine  participants.  Some  participants  needed  considerable 
prodding  to  get  them  started  thinking  futuristically.  Many  participants  were  not  particularly 
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aware  of  Future  Force  development  directions.  They  are  extremely  involved  in  their  day-to-day 
activities  and  have  limited  time  to  keep  up  with  planning  that  is  occurring  for  10  to  15  years 
down  the  road.  One  might  assume  that  these  would  be  less  productive  participants,  however,  it 
may  also  be  true  that  those  most  familiar  with  the  specifics  of  Future  Forces  development  efforts 
may  be  the  least  likely  to  provide  useful  information.  Once  some  of  the  ‘naive’  participants  got 
started,  some  were  highly  capable  of  thinking  beyond  their  own  current  assignments  and  view  of 
the  world.  On  the  other  hand,  one  of  the  participants  directly  involved  with  transition  to  the 
Future  Force  was  unable  to  overcome  the  specific  direction,  and  the  specific  details,  of  his 
knowledge  of  Future  Force  developments.  He  spent  most  of  his  time  explaining  the  concepts 
described  in  current  Army  planning  documents  (i.e.,  the  party  line)  and  could  not  be  induced  to 
think  critically  about  them.  Although  the  tools  were  designed  for  concept  elicitation,  they  may 
also  have  value  in  selecting  people  for  assignments  that  require  evaluation  of  future  concepts. 

In  the  current  project,  researchers  had  a  great  deal  of  latitude  to  select  scenarios  -  they 
simply  had  to  relate  to  the  See  function.  In  the  future,  the  Army  will  want  to  use  these  tools  to 
answer  specific  questions  that  arise  from  different  situations.  For  example,  as  the  “future 
thinkers”  attempt  to  predict  how  individual  Soldiers  and  units  will  operate,  they  will  raise 
questions  that  are  best  answered  empirically.  The  tools  provide  at  least  one  approach.  Research 
is  needed  to  show  how  well  the  methods  work  when  applied  to  specific  new  issues.  Some 
answers  to  this  question  may  be  found  in  a  related  ongoing  project  in  which  similar  tools  and 
methods  are  being  used  to  develop  training  for  a  task  that  lacks  recognized  experts  -  interagency 
crisis  action  planning  and  execution. 

How  can  we  best  use  the  total  Army  team  -  warfighter,  scientist,  theorist,  analyst,  and 
engineer  -  in  the  concept  development  process?  The  tools  described  here  are  a  preliminary  step 
in  answering  that  question.  They  represent  something  between  simple  “brainstorming”  and  full¬ 
blown  simulation  -  providing  structure  and  control  without  requiring  exorbitant  resource 
commitments.  What  other  effective  tools  fall  somewhere  in  between  those  two  extremes?  What 
is  the  best  way  to  use  the  SWE,  CDS,  and  other  new  tools  in  the  concept  discovery,  development, 
and  testing  process?  These  remain  open  questions  and  are  the  subject  of  continuing  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  research. 
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Appendix  A 
List  of  Acronyms 


AO 

ARI 

AWE 

BCT 

BDA 

Bde 

Bn 

CA 

CAB 

CCIR 

CDS 

C4ISR 

COA 

COP 

CTC 

DOD 

DOTMLPF 


FCS 

FE 

GCM 

HQ 

IPB 

ISG 

ISR 

JCS 

MI 

MTW 

NIMA 


Area  of  Operation 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
Advanced  Warfighter  Experiment 

Brigade  Combat  Team 
Battle  Damage  Assessment 
Brigade 
Battalion 

Combat  Assessment 
Combat  Aviation  Brigade 
Commander’s  Critical  Information  Requirement 
Concept-Development  Sessions 

Command,  Control,  Communications,  Computers,  Intelligence,  Surveillance  and 
Reconnaissance 
Course  of  Action 
Common  Operational  Picture 
Combat  Training  Center 

Department  of  Defense 

Doctrine,  Organization,  Training,  Material,  Leadership  and  Education,  Personnel, 
and  Facilities  (DOTMLPF) 

Future  Combat  Systems 
Formative  Evaluation 

Graphic  Control  Measures 

Headquarters 

Intelligence  Preparation  of  the  Battlespace 

Intuitive  Summary  Graphics 

Intelligence,  Surveillance,  and  Reconnaissance 

Joint  Chiefs  of  Staff 

Military  Intelligence 
Major  Theater  War 

National  Imagery  and  Mapping  Agency 
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o&o 

ORD 

RSTA 

SME 

SSC 

SWE 

Tip 

UA 

UAMBL 

UAV 

UE 

UGV 

WMD 


Operational  and  Organizational 
Operational  Requirements  Document 

Reconnaissance,  Surveillance,  and  Target  Acquisition 

Subject  Matter  Expert 
Small  Scale  Contingency 
Scaled-World  Events 

Tactics,  Techniques,  and  Procedures 

Unit  of  Action 

Unit  of  Action  Maneuver  Battle  Lab 
Unmanned  Aerial  Vehicle 
Unit  of  Employment 
Unmanned  Ground  Vehicle 

Weapons  of  Mass  Destruction 
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Appendix  B 

Scaled-World  Event  Specifications 


SCALED-WORLD  EVENT  1  SPECIFICATION 


Title  Detect  and  identify  threat  forces 

In  this  event,  participants  attempt  to  locate  and  identify  threat  forces  within 
their  area  of  operations  (AO)  by  deploying  sensors  and/or  obtaining 
information  from  sensors  owned  by  higher  or  adjacent  units.  The  event  is 
triggered  by  a  message  from  higher  stating  that  reports  of  possible  threat 
forces  in  their  AO  have  been  received. 

This  event  is  for  the  Unit  of  Action  (UA)  command  group  (with  two  members 
-  the  commander  and  the  intelligence,  surveillance,  and  reconnaissance  [ISR] 
Coordinator  -  participating). 

This  event  is  very  basic  and  serves  the  function  of  orienting  participants  to  the 
scaled-world  environment  as  well  as  providing  a  research  opportunity.  The 
only  variables  that  are  manipulated  in  this  event  are  the  number  and  types  of 
organic  sensors  available  to  the  unit,  and  the  numbers  and  types  of  non- 
organic  sensors  from  which  the  unit  can  obtain  information. 

This  event  measures  performance  on  the  dependent  variables  as  a  function  of 
manipulating  the  independent  variables;  however,  it  also  examines  process 
data.  That  is,  it  records  and  analyzes  the  specific  activities,  decisions,  and 
other  process  data  to  the  extent  possible  in  order  to  provide  a  more  complete 
understanding  of  the  measured  relationships  between  the  independent  and 
dependent  variables. 

Participants  begin  the  event  with  a  terrain  map  showing  their  AO.  They 
receive  a  message  informing  them  that  threat  force  movement  in  an  adjacent 
AO  was  detected  with  the  threat  possibly  moving  into  their  AO.  This 
provides  the  cue  to  conduct  electronic  surveillance  to  detect  and  identify  the 
possible  threat  forces. 

Threat  forces  consist  of  a  scout  platoon.  They  are  stationary  and 
camouflaged. 

The  event  is  conducted  during  daylight  conditions.  Skies  are  clear  with  no 
obstacles  to  visibility  related  to  weather. 

The  event  concludes  when  the  unit  detects,  locates  and  identifies  the  threat 
force. 


General 

description 


Tasks 


Decide  what  to  look  for. 
Decide  where  to  look. 
Decide  what  to  look  with. 
Review  information. 
Interpret  information. 
Evaluate  information. 
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Independent 

variables 

■  Number  and  type  of  organic  sensors. 

■  Number  and  type  of  non-organic  sensors. 

Dependent 

variables 

■  Time  to  detect  threat  forces. 

■  Accuracy  of  detection  and  identification. 

Process  data 

■  Observations/data  of  participant  performance/actions  during 

completion  of  the  event,  collected  automatically. 

*  Participant  responses/reactions,  collected  during  post-event  debriefing. 

Measurement 

plan 

The  two  dependent  variables  (time  required  to  locate  and  identify  threat  forces 
and  accuracy  of  location  and  identification)  are  measured  by  observation. 

Two  observers  monitor  each  event  and  record  start-time  and  end-time.  The 
observers  also  record  the  accuracy  of  the  location  and  identification  of  the 
threat  as  either  correct  or  incorrect.  These  measures  provide  the  primary  basis 
for  comparing  the  effects  of  manipulating  the  independent  variables  of  types 
and  numbers  of  organic  and  non-organic  sensors  available  to  the  participants. 

In  addition,  the  scaled-world  environment  is  constructed  to  collect  this  data  to 
the  extent  possible. 

In  addition  to  these  measures,  the  complete  event  session  is  videotaped  in 
order  to  capture  process  data.  The  observers  and  the  scaled- world 
environment  also  record  process  data  as  the  event  proceeds.  They  record  the 
steps/actions  the  participants  used  in  deploying  sensors,  in  processing 
information  received  from  the  sensors,  data  received  from  the  sensors, 
decisions  made,  interactions  between  participants,  and  other  process 
information  as  appropriate.  The  observers  employ  an  observation  checklist. 
However,  it  will  not  be  used  restrictively.  The  tapes  of  the  session  along  with 
the  data  collected  by  the  scaled-world  environment  are  used  to 
verify/supplement  observer  recorded  process  data. 

Following  the  event,  the  participants  are  debriefed  by  the  observers  in  a 
structured  interview  setting.  The  observers  use  a  structured  interview  form  for 
conducting  the  debrief  session  and  for  recording  responses.  These  sessions 
are  also  videotaped  to  allow  verification  of  observer-recorded  data. 

Data  analysis 

Primary  data  analysis  consists  of  (1)  comparing  the  time  to  locate  and  identify 
the  threat  force  as  a  function  of  numbers  and  types  of  organic  and  non-organic 
sensors;  and  (2)  comparing  the  accuracy  of  location  and  identification  of  the 
threat  force  as  a  function  of  numbers  and  types  of  organic  and  non-organic 
sensors. 

Secondary  data  analysis  consists  of  performing  qualitative  analysis  of  the 
process  data  recorded  during  the  event  and  post-event  debriefing.  This 
analysis  is  conducted  to  provide  further  explanation  of  results  observed  in  the 
primary  analysis. 
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SCALED-WORLD  EVENT  2  SPECIFICATION 


Title 

General 

description 


Tasks 


Locate  a  mobile  based  biological  weapon  threat 

The  enemy  has  a  biological  weapon  of  mass  destruction  (WMD)  that  is 
contained  on  a  mobile  platform  (probably  a  medium  to  large  truck).  The 
objective  of  the  event  is  to  locate  the  truck  and  the  weapon.  The  unit  has  two 
types  of  sensors  that  are  capable  of  detecting  biological  weapons:  a  fully 
robotic  ground-based  sensor  capable  of  detecting  a  biological  weapon  from  a 
distance  less  than  100  meters;  and  a  manned  sensor  capable  of  detecting  a 
biological  weapon  from  a  distance  less  than  10  meters.  Both  sensors  have 
error  rates  of  25%  false  positives  and  1%  false  negatives.  The  unit  has 
organic  ground  and  air  sensors  capable  of  providing  visual  (video,  infrared, 
radar)  images  and  access  to  information  from  non-organic  sensors.  They  also 
have  ground  sensors  capable  of  providing  audio  and  seismic  data. 

This  event  is  for  the  UA  command  group  (with  two  members  -  the 
commander  and  the  ISR  Coordinator  -  participating). 

This  event  is  process  oriented.  The  primary  interest  is  in  observing  the 
strategy  that  the  group  employs  in  locating  the  biological  weapon.  Because  of 
the  limited  range  of  the  biological  weapons  sensors,  the  group  will  need  to 
develop  a  strategy  for  using  its  other  reconnaissance  and  surveillance  assets  to 
isolate  probable  locations  of  the  platform  containing  the  biological  weapon. 

Participants  begin  the  event  with  a  terrain  map  showing  their  AO.  They 
receive  a  message  informing  them  that  intelligence  reports  indicate  that 
enemy  forces  are  thought  to  have  a  biological  weapon  which  has  been  placed 
on  an  unknown  mobile  platform.  The  location  of  the  platform  and  weapon  is 
unknown;  however,  its  detection  and  identification  are  highest  priority. 

The  event  is  conducted  during  daylight  conditions.  Skies  are  clear  with  no 
obstacles  to  visibility  related  to  weather. 

The  event  concludes  when  the  biological  weapon  has  been  located. 

■  Decide  where  to  look. 

*  Decide  what  to  look  with. 

■  Fuse  information  on  screens. 

■  Review  information. 

■  Interpret  information. 

■  Evaluate  information. 

■  Verify  information. 


Independent  None  (Note:  Future  variations  of  this  event  could  include  manipulation  of 
variables  variables  such  as  weather  conditions,  time  of  day/night,  range  of  biological- 

sensors) 


Dependent 

variables 

■  Time  to  locate  the  biological  weapon. 

■  Accuracy  of  detection. 

Process  data 

■  Observations/data  of  participant  performance/actions  during 
completion  of  the  event,  collected  automatically. 

■  Participant  responses/reactions,  collected  during  post-event  debriefing. 

Measurement 

plan 

The  two  dependent  variables  (time  required  to  locate  and  accuracy  of 
detection)  are  measured  by  observation.  Two  observers  monitor  each  event 
and  record  start-time  and  end-time.  The  observers  also  record  the  accuracy  of 
the  participants’  location  and  identification  as  either  correct  or  incorrect.  In 
addition,  the  scaled- world  environment  is  constructed  to  collect  this  data  to 
the  extent  possible. 

In  addition  to  these  measures,  the  complete  event  session  is  videotaped  in 
order  to  capture  process  data  which,  as  stated  above,  are  the  date  of  primary 
interest  for  this  event.  The  observers  also  record  process  data  as  the  event 
proceeds.  They  record  the  steps/actions  the  participants  used  in  deploying 
sensors,  in  processing  information  received  from  the  sensors,  data  received 
from  the  sensors,  decisions  made,  interactions  between  participants,  and  other 
process  information  as  appropriate.  The  observers  employ  an  observation 
checklist;  however,  it  is  not  to  be  used  restrictively.  The  tapes  of  the  session 
along  with  the  data  collected  by  the  scaled-world  environment  are  used  to 
verify/supplement  observer  recorded  process  data. 

Following  the  event,  the  participants  are  debriefed  by  the  observers  in  a 
structured  interview  setting.  The  observers  use  a  structured  interview  form 
for  conducting  the  debrief  session  and  for  recording  responses.  These 
sessions  are  also  videotaped  to  allow  verification  of  observer-recorded  data. 

Data  analysis 

Primary  data  analysis  consists  of  performing  qualitative  analysis  of  the 
process  data  recorded  during  the  event  and  post-event  debriefing.  This 
analysis  attempts  to  identify  factors  in  the  participants  performance  related  to 
effective  and  ineffective  results  (as  indicated  by  successful  location  of  the 
biological  weapon  in  a  timely  manner).  The  analysis  also  seeks  to  identify 
factors/variables  which  would  have  facilitated  performance  by  examining 
results  obtained  from  the  participant  debriefing  session. 
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SCALED-WORLD  EVENT  3  SPECIFICATION 


Title  What  course  of  action  is  the  enemy  taking? 


The  unit  has  received  a  message  from  UA  Battalion  showing  likely  courses  of 
action  (COA)  the  enemy  may  take.  A  number  of  these  COAs  fall  within  the 
unit’s  AO.  The  unit  is  directed  to  initiate  reconnaissance  and  surveillance  to 
determine  which  COAs  the  enemy  is  actually  taking  and  to  report  the 
information  to  Battalion. 

This  event  is  for  the  UA  command  group  (with  two  members  -  the 
commander  and  the  ISR  Coordinator  -  participating). 

This  event  measures  performance  on  the  dependent  variables  as  a  function  of 
manipulating  the  independent  variables;  however,  it  also  examines  process 
data.  That  is,  it  records  and  analyzes  the  specific  activities,  decisions,  and 
other  process  data  to  the  extent  possible  in  order  to  provide  a  more  complete 
understanding  of  the  measured  relationships  between  the  independent  and 
dependent  variables. 

This  event  provides  an  opportunity  to  exercise  more  of  the  “SEE”  tasks  than 
does  the  “Detect  and  identify  threat  forces”  event,  and  also  introduces 
additional  dependent  and  independent  variables.  Independent  variables 
include  numbers  and  types  of  sensors,  numbers  of  COAs  lying  within  the 
Company’s  AO,  number  of  COAs  the  enemy  actually  follows,  and 
unanticipated  COAs.  Dependent  variables  include  time  required  to  determine 
enemy  COAs  and  accuracy  of  determining  enemy  COAs. 

Participants  begin  the  event  with  a  terrain  map  showing  their  AO.  They 
receive  a  message  from  Battalion  Headquarters  (HQ)  detailing  selected 
intelligence  preparation  of  the  battlespace  (IPB)  products  including  potential 
enemy  COAs  and  directing  them  to  conduct  reconnaissance  and  surveillance 
to  identify  which  COAs  the  enemy  is  following. 

The  event  is  conducted  during  daylight  conditions.  Skies  are  clear  with  no 
obstacles  to  visibility  related  to  weather. 

The  event  concludes  when  the  Company  command  group  reports  enemy 
COA(s)  to  Battalion  HQ. 


General 

description 
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Tasks 

■  Decide  where  to  look. 

■  Decide  what  to  look  with. 

■  Fuse  information  on  screens. 

■  Review  information. 

■  Interpret  information. 

■  Evaluate  information. 

■  Verify  information. 

■  Track  forces. 

•  Share  information. 

Independent 

variables 

■  Number  and  type  of  organic  sensors. 

■  Number  and  type  of  non-organic  sensors. 

■  Number  of  CO  As  lying  within  the  Company’s  AO. 

■  Number  of  CO  As  the  enemy  follows. 

■  Unanticipated  COAs  the  enemy  follows. 

Dependent 

variables 

■  Time  to  identify  enemy  COA  after  enemy  initiates  activity. 

■  Accuracy  of  identification. 

*  Timeliness  and  accuracy  of  reporting  to  Battalion  HQ. 

Process  data 

*  Observations/data  of  participant  performance/actions  during 

completion  of  the  event,  collected  automatically. 

■  Participant  responses/reactions,  collected  during  post-event  debriefing. 

Measurement 

plan 

The  dependent  variables  are  measured  by  observation.  Two  observers 
monitor  each  event  and  record  start-time,  time  each  enemy  COA  was 
identified,  time  reports  were  made  to  Battalion  HQ,  and  end-time.  The 
observers  also  record  the  accuracy  of  the  participants’  identification  as  either 
correct  or  incorrect.  In  addition,  the  scaled-world  environment  is  constructed 
to  collect  this  data  to  the  extent  possible. 

In  addition  to  these  measures,  the  complete  event  session  is  videotaped  in 
order  to  capture  process  data.  The  observers  also  record  process  data  as  the 
event  proceeds.  They  record  the  steps/actions  the  participants  used  in 
deploying  sensors,  in  processing  information  received  from  the  sensors,  data 
received  from  the  sensors,  decisions  made,  interactions  between  participants, 
communication  with  Battalion  HQ,  and  other  process  information  as 
appropriate.  The  observers  employ  an  observation  checklist;  however,  it  is 
not  used  restrictively.  The  tapes  of  the  session  along  with  the  data  collected 
by  the  scaled- world  environment  are  used  to  verify  and  supplement  observer 
recorded  process  data. 

Following  the  event,  the  participants  are  debriefed  by  the  observers  in  a 
structured  interview  setting.  The  observers  use  a  structured  interview  form  for 
conducting  the  debrief  session  and  for  recording  responses.  These  sessions 
are  also  videotaped  to  allow  verification  of  observer-recorded  data. 
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Data  analysis  Primary  data  analysis  consists  of  (1)  comparing  the  accuracy  of  CO  A 

identification  (i.e.,  the  COA(s)  the  enemy  actually  follows)  as  a  function  of 
numbers  and  types  of  organic  and  non-organic  sensors,  numbers  of  COAs 
falling  within  the  AO,  number  of  COAs  actually  followed,  and  unanticipated 
COA(s)  followed;  (2)  comparing  the  time  required  to  identify  COA(s) 
followed  as  a  function  of  these  variables;  and  (3)  accuracy  and  timeliness  of 
reporting  to  Battalion  HQ  as  a  function  of  these  variables. 

Secondary  data  analysis  consists  of  qualitative  analysis  of  the  process  data 
recorded  during  the  event  and  post-event  debriefing.  This  analysis  is 
conducted  to  provide  further  explanation  of  results  observed  in  the  primary 
analysis. 
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SCALED-WORLD  EVENT  4  SPECIFICATION 


Title  Party  or  battle? 

General  U.S.  forces  are  in  Alkastan  as  part  of  an  allied  peace  keeping  operation. 

description  Alkastan  has  been  engaged  in  a  civil  war  based,  for  the  most  part,  upon  ethnic 

divisions  within  the  country.  Neither  faction  is  particularly  friendly  toward 
allied  forces,  and  there  continue  to  be  numerous  skirmishes  between  factions. 
Allied  forces  routinely  conduct  reconnaissance  and  surveillance  using  air  and 
ground  assets,  both  manned  and  robotic,  and  on  occasion,  these  forces  come 
under  fire.  On  the  other  hand,  there  have  been  two  recent  incidents  in  which 
allied  forces  incorrectly  identified  actions  of  civilian  populations  as 
threatening,  firing  on  them  and  producing  casualties.  Allied  forces  have 
concluded  that,  from  a  distance,  civilian  celebrations  are  difficult  to 
distinguish  from  threatening  actions  since  both  involve  small-arms  fire,  and  a 
lot  of  loud  vocalizations.  In  the  present  instance,  a  reconnaissance, 
surveillance,  and  target  acquisition  (RSTA)  troop  has  been  cued  by  a  brigade 
sensor  to  investigate  a  group  of  about  100  people  shouting  and  firing 
weapons.  The  troop  commander  must  determine  whether  this  represents  a 
threat  to  his  unit. 

This  event  is  for  the  troop  commander. 

The  troop  has  sensors  and  scouts  capable  of  sending  live  video  to  the  troop 
commander  as  well  as  to  higher.  It  also  has  organic  unmanned  aerial  vehicles 
(UAVs)  capable  of  providing  video  and  infrared  images.  It  has  unmanned 
ground  vehicles  (UGVs)  that  can  provide  seismic  information.  The  unit 
leader  has  no  access  to  any  additional  sensor  information. 

Two  independent  variables  are  manipulated  in  this  event:  the  time  of  day 
(daylight  or  dark)  and  the  threat  level  (no  threat  -  it’s  a  celebration;  threat  to 
other  ethnic  group;  threat  to  the  unit).  The  dependent  variable  is  identification 
(correct  or  incorrect)  of  the  threat  level.  This  event  measures  performance  on 
the  dependent  variable  as  a  function  of  manipulating  the  independent 
variables;  however,  it  also  examines  process  data.  That  is,  it  records  and 
analyzes  the  specific  activities,  decisions,  and  other  process  data  to  the  extent 
possible  in  order  to  provide  a  more  complete  understanding  of  the  measured 
relationships  between  the  independent  and  dependent  variables. 

Participants  begin  the  event  upon  being  cued  by  sensor  feed.  The  troop 
commander  must  determine  where  the  fire  is  originating  and  the  probable 
intent  of  the  group  initiating  fire. 
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Tasks  *  Position  sensors/soldiers. 

■  Review  information. 

•  Interpret  information. 

•  Evaluate  information. 

■  Verify  information. 

*  Share  information. 

Independent  ■  Time  of  day  (daylight  or  night). 

variables  ■  Threat  condition  (no  threat;  threat  against  another  force;  threat  against 

the  platoon). 

Dependent  *  Accuracy  of  identification  of  threat  condition. 

variable 

Process  data  ■  Observations/data  of  participant  performance/actions  during 

completion  of  the  event,  collected  automatically. 

*  Participant  responses/reactions,  collected  during  post-event  debriefing. 

Measurement  The  dependent  variable  is  measured  by  observation.  Two  observers  monitor 

plan  each  event  and  record  the  threat  condition  identified.  The  observers  also 

record  the  accuracy  of  the  participants’  identification  as  either  correct  or 
incorrect.  In  addition,  the  scaled-world  environment  is  constructed  to  collect 
data  to  the  extent  possible. 

In  addition  to  these  measures,  the  complete  event  session  is  videotaped  in 
order  to  capture  process  data.  The  observers  also  record  process  data  as  the 
event  proceeds.  They  record  the  steps/actions  the  participants  used  in 
deploying  personnel  and  sensors,  in  processing  information  received  from  the 
personnel  and  sensors,  data  received  from  the  personnel  and  sensors, 
decisions  made,  interactions  between  participants,  communication  with  higher 
HQ,  and  other  process  information  as  appropriate.  The  observers  employ  an 
observation  checklist;  however,  it  is  not  used  restrictively.  The  tapes  of  the 
session  along  with  the  data  collected  by  the  scaled-world  environment  are 
used  to  verify/supplement  observer  recorded  process  data. 

Following  the  event,  the  participants  are  debriefed  by  the  observers  in  a 
structured  interview  setting.  The  observers  use  a  structured  interview  form 
for  conducting  the  debrief  session  and  for  recording  responses.  These 
sessions  are  also  videotaped  to  allow  verification  of  observer-recorded  data. 


Data  analysis  Primary  data  analysis  consists  of  comparing  the  accuracy  of  the  threat 

condition  identification  as  a  function  of  the  actual  threat  condition  and  time  of 
day. 

Secondary  data  analysis  consists  of  performing  qualitative  analysis  of  the 
process  data  recorded  during  the  event  and  post-event  debriefing.  This 
analysis  is  conducted  to  provide  further  explanation  of  results  observed  in  the 
primary  analysis. 
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SCALED-WORLD  EVENT  5  SPECIFICATION 


Title  Sensor  overload? 


General 

description 


Tasks 


The  commander  of  the  Military  Intelligence  (MI)  Company  has  been  directed 
by  Brigade  HQ  to  monitor  sensor  feeds  and  report  when  critical  event  XX 
occurs  (e.g.,  a  column  of  heavy  vehicles  passes  a  specific  location.)  The 
commander  has  feeds  from  sensors  that  detect  and  feed  video,  infrared,  and 
seismic  signals.  The  task  becomes  more  difficult  as  the  number  and  types  of 
available  feeds  increases,  and  as  the  number  of  events  the  commander  is 
monitoring  for  increases. 

This  event  is  for  the  MI  Company  commander. 

This  event  is  designed  to  examine  the  processing  capabilities  of  an  observer 
as  a  function  of  the  number  and  type  of  information  streams  being  observed, 
and  of  the  number  of  targets  the  observer  is  attempting  to  detect.  Three 
independent  variables  are  manipulated  in  this  event:  the  number  of  sensor 
feeds  being  monitored;  the  type  of  sensor  feeds  being  monitored;  and  the 
number  of  critical  events  the  participant  is  monitoring  for.  Two  dependent 
variables  are  measured:  was  the  critical  event  detected;  and  how  long  after  it 
first  appeared  in  a  sensor  feed  was  the  event  detected.  This  event  measures 
performance  on  the  dependent  variables  as  a  function  of  manipulating  the 
independent  variables;  however,  it  also  examines  process  data.  That  is,  it 
records  and  analyzes  the  specific  activities,  decisions,  and  other  process  data 
to  the  extent  possible  in  order  to  provide  a  more  complete  understanding  of 
the  measured  relationships  between  the  independent  and  dependent  variables. 

Because  of  the  likelihood  of  data  confounding  if  a  repeated  measures  design 
were  employed,  each  participant  participates  in  a  single  trial.  Participants 
begin  the  event  upon  receiving  an  order  from  Brigade  (Bde)  HQ  to  observe 
sensor  feeds  1  -  n  for  the  occurrence  of  critical  events  a  -  x.  The  number  of 
sensor  feeds  varies  from  1  to  9.  The  type  varies  among  video,  infrared,  and 
seismic.  The  display  size  for  each  sensor  remains  constant  regardless  of  the 
number  of  sensor  feeds  employed.  The  number  of  critical  events  varies  from 
1  to  3.  (Note:  the  participant  does  not  control  sensor  placement  or  type;  he  is 
monitoring  sensor  feeds.  In  addition,  in  situations  where  there  is  a  single 
sensor  feed,  the  critical  event  will  be  one  appropriate  to  the  type  of  sensor 
being  employed).  As  soon  as  a  critical  event  is  detected,  the  participant 
reports  to  Battalion  (Bn)  HQ  via  radio  contact. 

■  Filter  sensor  feeds. 

■  Review  information. 

■  Interpret  information. 

■  Evaluate  information. 

■  Verify  information. 

■  Share  information. 
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Independent 

variables 

■  Number  of  sensor  feeds. 

■  Type  of  sensor  feeds. 

■  Number  of  critical  events. 

Dependent 

■  Accuracy  of  critical  event  detection. 

variable 

■  Time  required  to  detect  critical  event. 

Process  data 

■  Observations/data  of  participant  performance/actions  during 
completion  of  the  event,  collected  automatically. 

■  Participant  responses/reactions,  collected  during  post-event  debriefing. 

The  dependent  variables  are  measured  by  observation.  Two  observers 
monitor  each  event  and  record  when  the  sensor  feed  indicating  the  occurrence 
of  a  critical  event  occurred.  They  also  record  if,  and  when,  the  participant 
observed  the  critical  event  and  made  his  report  to  HQ.  This  process  is 
repeated  for  each  critical  event  included. 

In  addition  to  these  measures,  the  complete  event  session  is  videotaped  in 
order  to  capture  process  data.  The  observers  also  record  process  data  as  the 
event  proceeds.  They  record  the  steps/actions  the  participants  used  in 
deploying  personnel  and  sensors,  in  processing  information  received  from  the 
personnel  and  sensors,  data  received  from  the  personnel  and  sensors,  decisions 
made,  interactions  between  participants,  communication  with  Company  HQ, 
and  other  process  information  as  appropriate.  The  observers  employ  an 
observation  checklist;  however,  it  will  not  be  used  restrictively.  The  tapes  of 
the  session  along  with  the  data  collected  by  the  scaled-world  environment  are 
used  to  verify/supplement  observer  recorded  process  data. 

Following  the  event,  the  participants  are  debriefed  by  the  observers  in  a 
structured  interview  setting.  The  observers  use  a  structured  interview  form  for 
conducting  the  debrief  session  and  for  recording  responses.  These  sessions 
are  also  be  videotaped  to  allow  verification  of  observer-recorded  data. 

Data  analysis  Primary  data  analysis  consists  of  comparing  the  accuracy  and  latency  of  the 
critical  event  detection  across  participants  as  a  function  of  the  number  of 
sensor  feeds,  type  of  sensor  feeds,  and  number  of  critical  events. 

Secondary  data  analysis  consists  of  performing  qualitative  analysis  of  the 
process  data  recorded  during  the  event  and  post-event  debriefing.  This 
analysis  is  conducted  to  provide  further  explanation  of  results  observed  in  the 
primary  analysis. 


Measurement 

plan 
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SCALED-WORLD  EVENT  6  SPECIFICATION 


Title 

Determine  battle  damage  assessment 

General 

description 

Friendly  forces  have  conducted  an  air  assault  in  A  Company’s  AO.  Aircraft 
will  continue  to  be  in  a  position  to  conduct  additional  strikes  based  on  battle 
damage  assessment  (BDA)  completed  by  A  company. 

This  event  is  for  the  UA  command  group  (with  two  members  -  the 
commander  and  the  ISR  Coordinator  -  participating). 

This  event  is  process  oriented.  The  primary  interest  is  in  observing  the 
strategy  that  the  group  employs  in  assessing  the  damage  produced  by  the  air 
strikes.  Because  the  time  to  conduct  BDA  is  limited,  the  group  needs  to 
develop  a  strategy  for  using  its  sensor  assets  as  efficiently  as  possible. 

Participants  begin  the  event  with  a  terrain  map  showing  their  AO.  They 
receive  a  message  directing  them  to  assess  damage  produced  by  air  strikes  at 
three  locations  in  their  AO.  They  are  given  a  time  limit  to  report  BDA  results 
based  on  how  long  the  attack  aircraft  will  be  in  position  to  initiate  another 
attack. 

The  event  is  conducted  during  daylight  conditions.  Skies  are  clear  with  no 
obstacles  to  visibility  related  to  weather. 

The  event  concludes  with  the  BDA  report. 

Tasks 

■  Decide  what  to  look  with. 

•  Fuse  information  on  screens. 

■  Review  information. 

■  Interpret  information. 

■  Evaluate  information. 

■  Verify  information. 

Independent 

variables 

■  Time  attack  aircraft  will  be  in  position  to  initiate  a  second  attack  (15 
or  30  minutes). 

Dependent 

variable 

■  Accuracy  of  BDA . 

■  Time  to  conduct  BDA. 

Process  data 

■  Observations/data  of  participant  performance/actions  during 

completion  of  the  event,  collected  automatically. 

*  Participant  responses/reactions,  collected  during  post-event  debriefing. 
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The  two  dependent  variables  (accuracy  of  BDA  and  time  to  conduct  BDA)  are 
measured  by  observation.  Two  observers  monitor  each  event  and  record  start¬ 
time  and  end-time.  The  observers  also  record  the  accuracy  of  the  participants’ 
assessment  as  either  correct  or  incorrect.  In  addition,  the  scaled- world 
environment  is  constructed  to  collect  this  data  to  the  extent  possible. 

In  addition  to  these  measures,  the  complete  event  session  is  videotaped  in 
order  to  capture  process  data  which,  as  stated  above,  are  the  date  of  primary 
interest  for  this  event.  The  observers  also  record  process  data  as  the  event 
proceeds.  They  record  the  steps/actions  the  participants  used  in  deploying 
sensors,  in  processing  information  received  from  the  sensors,  data  received 
from  the  sensors,  decisions  made,  interactions  between  participants,  and  other 
process  information  as  appropriate.  The  observers  employ  an  observation 
checklist;  however,  it  will  not  be  used  restrictively.  The  tapes  of  the  session 
along  with  the  data  collected  by  the  scaled-world  environment  are  used  to 
verify/supplement  observer  recorded  process  data. 

Following  the  event,  the  participants  are  debriefed  by  the  observers  in  a 
structured  interview  setting.  The  observers  use  a  structured  interview  form 
for  conducting  the  debrief  session  and  for  recording  responses.  These 
sessions  are  also  videotaped  to  allow  verification  of  observer-recorded  data. 


Data  analysis  Primary  data  analysis  consists  of  qualitative  analysis  of  the  process  data 

recorded  during  the  event  and  post-event  debriefing.  This  analysis  attempts  to 
identify  factors  in  the  participants  performance  related  to  effective  and 
ineffective  results  (as  indicated  by  successful  location  of  the  biological 
weapon  in  a  timely  manner).  The  analysis  also  seeks  to  identify 
factors/variables  which  would  have  facilitated  performance  by  examining 
results  obtained  from  the  participant  debriefing  session. 


Measurement 

plan 
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Appendix  C 

Scaled-World  Environment 
User  Instructions 
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Scaled-World  Environment 


User  Instructions 


I.  Loading  A  Scenario 

1.  After  deciding  which  scenario  you  will  execute,  click  the  left  mouse  button  on  the 
Scenario  menu  and  move  the  mouse  over  the  selected  scenario.  As  shown  in  Figure  C-l. 
Next  select  the  desired  version  of  the  scenario  from  the  submenu,  as  shown  in  Figure  C-2. 
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Figure  C-l .  Scenario  selection  dropdown  menu. 
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Figure  C-2.  Scenario  version  dropdown  menu. 

2.  When  the  scenario  is  loaded,  a  description  of  the  scenario  situation  is  displayed.  This 
description  provides  details  of  the  current  situation  as  well  as  a  list  of  the  sensors  you  have 
available.  After  reading  the  situation,  you  may  close  the  window  by  clicking  the  Close 
button.  At  any  time  during  the  scenario,  you  can  view  the  details  again  by  clicking  on 
Situation  Details  in  the  lower  left  hand  comer  of  the  screen.  See  Figure  C-3 


Situation 

Situation  Detail;! 


Figure  C-3.  Situation  details  button. 
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II.  Screen  Layout  -  When  the  scenario  loads,  your  screen  will  appear  similar  to  the  one  shown  in 
Figure  C-4.  The  screen  includes  four  main  panels: 


Figure  C-4.  Initial  scaled- world  tool  screen  layout. 

1 .  Operations  Panel  -  on  the  left  side  of  the  screen;  changes  depending  upon  which  panel  is 
selected  (see  below).  The  operations  panel  is  used  to  interact  with  and  obtain 
information  from  the  map,  deploy  sensors,  and  select  sensors  to  view. 

2.  Panel  Select  Buttons  -  across  the  top  of  the  screen;  control  the  current  operation  panel. 

3.  Map  Area  -  the  map  area  displays  the  current  map  view,  along  with  any  sensors  that 
might  be  within  the  displayed  region. 

4.  Sensor  Output  Area  -  across  the  bottom  of  the  screen;  displays  text  messages  sent  by 
deployed  sensors. 
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III.  Navigating  the  Map 


1 .  The  navigation  panel  is  the  default  panel  displayed  in  the  Operations  Panel.  If  you  have 
selected  the  New  Sensors  or  Current  Sensors  panel,  you  can  return  to  the  navigation 
panel  by  selecting  the  Map  Navigation  button. 

2.  Map  navigation  consists  of  two  functions:  Pan  and  Zoom. 

•  The  pan  buttons  are  the  directional  arrows  in  the  upper  part  of  the  Navigation  panel. 
To  move  the  map,  select  the  appropriate  arrow.  The  amount  of  movement  depends 
on  the  current  area  of  the  displayed  map.  In  the  center  of  the  arrows  is  a  square  block. 
Clicking  this  block  will  return  the  center  of  the  map  to  the  center  of  the  screen. 

•  The  zoom  buttons  are  the  magnifying  glass  images  with  the  plus  (+)  and  minus  (-) 
signs.  When  clicked,  the  plus  image  zooms  in  (displays  a  smaller  area),  and  the 
minus  image  zooms  out  (displays  a  larger  area). 

IV.  Deploying  Sensors 

Sensors  are  deployed  using  the  New  Sensors  panel.  Deployment  involves  two  functions: 
selecting  the  type  of  sensor  you  want  to  deploy;  and  laying  down  (i.e.,  drawing)  the  sensor's 
route  of  travel.  However,  before  describing  these  functions,  you  need  to  know  some  basic 
sensor  information. 

1 .  Basic  Sensor  Platform  Information 

There  are  three  main  types  of  sensor  platforms:  Aerial,  Ground,  and  Stationary.  Each 
platform  is  capable  of  carrying  a  variety  of  sensor  types.  All  sensor  identification  codes 
start  with  “U”  indicating  that  they  are  unmanned. 

•  Aerial  -  air  based  platforms  for  which  you  can  set  a  speed  and  altitude.  These  sensors 
are  identified  by  the  “AV”  in  the  sensor  code  (i.e.,  UAV). 

•  Ground  -  land  based  platforms  for  which  you  can  set  a  speed.  These  sensors  are 
identified  by  the  “GV”  in  the  sensor  code  (i.e.,  UGV). 

•  Stationary  -  land  based  platforms  that  have  no  movement.  These  sensors  are 
identified  by  the  “GS”  in  the  sensor  code  (i.e.,  UGS). 

2.  Basic  Sensor  Types 

There  are  four  classes  of  sensors:  visual,  audio,  seismic,  and  biological  agent.  In 
general,  aerial  platforms  will  carry  visual  sensors;  land  based  platforms  will  carry  audio, 
seismic,  and  biological  agent.  There  is  also  a  hand-held  biological  agent  sensor. 

•  Visual  -  sensors  that  display  a  visual  image.  There  are  three  types  of  visual  sensors: 
video  (displays  a  basic  daylight  image),  infrared  (displays  image  filtered  through 
infrared  filters),  and  thermal  (displays  visual  image  of  heat  emissions  given  off  by 
objects). 
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•  Audio  -  sensors  that  detect  and  record  sounds. 

•  Seismic  -  sensors  that  detect  ground  vibrations  such  as  those  produced  by  tracked 
vehicles,  trucks,  etc. 

•  Biological  agent  -  sensors  that  detect  the  presence  of  biological  agents  such  as 
anthrax. 

3.  Selecting  a  Sensor  for  Deployment 

•  When  you  select  the  NewSensors  panel,  it  will  display  a  list  of  the  sensor  types 
(platform  and  sensor)  that  are  available  for  the  scenario.  Click  on  the  sensor  you  wish 
to  deploy. 

•  When  you  select  a  UAV  or  UGV  type  sensor,  a  sensor  properties  box  shown  in  Figure 
C-5,  lets  you  set  the  speed  and  (for  UAVs)  altitude  of  the  sensor.  After  setting  the 
sensor  parameters,  Click  the  Done  button.  You  can  now  lay  down  the  route  (see 
below). 

•  If  you  select  a  stationary  sensor,  you  will  not  see  a  sensor  control  box.  Stationary 
sensors  do  not  move  and,  therefore,  all  you  need  to  do  is  place  them. 


Sensor  Properties 


Sensor  Properties 


UAV-IR 


Speed  (kph) 


/  N  ,  : ’  !  '  - 


.  + 


Altitude  do 


. 


Done 


Figure  C-5.  Sensor  properties  control. 

4.  Laying  down  the  sensor's  route  of  travel. 

•  If  you  have  selected  a  UAV  or  UGV  for  deployment,  you  must  specify  its  route  of 
travel  after  you  have  specified  its  properties.  You  will  do  this  on  the  map  itself  by 
specifying  a  starting  point,  interim  waypoints,  and  a  final  loitering  pattern. 

•  To  select  a  starting  point,  move  the  mouse  pointer  to  the  desired  location  on  the  map 
and  Left  Click. 

•  Move  the  mouse  pointer  to  the  first  desired  waypoint  and  Left  Click. 

•  Repeat  step  3  for  each  desired  waypoint. 
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•  You  can  now  specify  a  circular  loitering  pattern  (i.e.,  a  pattern  the  sensor  will 
continue  to  move  in  for  the  remainder  of  the  scenario  or  until  you  recall  it.)  Move  the 
mouse  pointer  to  the  location  of  the  center  of  the  loitering  pattern  and  Right  Click. 

•  Move  the  mouse  pointer  and  a  circular  track  will  appear  on  the  map  as  shown  in 
Figure  C-6.  When  you  have  laid  down  the  desired  loitering  pattern,  Left  Click.  A 
green  square  will  appear  on  the  map  with  an  appropriate  sensor  ID. 

•  If  you  want  to  cancel  a  sensor  before  you  finish  its  deployment,  just  Double  Click  the 
left  mouse  button  in  the  Map  Area. 


Figure  C-6.  Loitering  sensor. 


V.  Viewing  Sensor  Output 


When  you  select  the  Current  Sensors  panel,  you  will  be  able  to  access  all  of  the  sensors 
you  have  deployed  in  addition  to  any  other  sensor  inputs  you  may  have  access  to.  For 
example,  you  may  be  able  to  access  the  feed  from  a  satellite  controlled  by  a  Unit  of 
Employment  (UE).  Each  of  the  sensor  types  has  its  own  output.  In  addition,  a  sensor  may 
display  a  text  message  in  the  Sensor  Output  panel.  You  can  scroll  through  all  of  the 
messages  that  have  been  sent  by  the  various  sensors  during  the  course  of  the  scenario.  There 
are  three  types  of  sensor  output:  audible,  visual,  and  textual.  Different  scenarios  require  the 
use  of  different  types  of  sensors.  Each  is  available  in  the  appropriate  scenario. 

1 .  Seismic  sensors  will  alert  the  user  of  activity  via  a  text  message  in  the  sensor  output 
area.  See  Figure  C-7.  Sensor  output  details  can  also  be  seen  by  selecting  the  sensor 
from  the  current  sensors  panel.  The  Bring  In  button  can  be  used  to  return  the  sensor. 
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Figure  C-7.  Audible  sensor  output. 


2.  Audible  sensors  will  alert  the  user  of  activity  in  the  sensor  output  area  via  a  text  message 
in  the  sensor  output  area.  To  listen  to  the  sound  recorded,  bring  up  the  current  sensors 
panel  and  click  the  Audible  Sensor.  Then  click  the  Play  button.  The  Bring  In  button 
can  be  used  to  return  the  sensor.  See  Figure  C-8. 


Figure  C-8.  Audible  sensor  controls. 

3.  Visual  sensors  provide  visual  images  that  are  only  accessible  through  the  current  sensor 
panel.  See  Figure  C-9.  To  view  the  images,  click  on  the  sensor  in  the  current  sensor 
panel.  While  the  sensor  output  window  is  open,  the  image  will  update  based  on  the 
position  of  the  sensor  in  the  map.  The  Bring  In  button  can  be  used  to  return  the  sensor. 
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Figure  C-9.  Visual  sensor  image. 


Biological  agent  sensors  will  alert  the  user  to  agents  detected  via  a  text  message  in  the 
sensor  output  area.  You  can  select  the  sensor  from  the  available  sensors  panel  to  view 
specific  details.  The  Bring  In  button  can  be  used  to  return  the  sensor. 
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CONCEPT-DEVELOPMENT  SESSION  1  SPECIFICATION 


Title  What  do  you  want  to  know,  how  do  you  want  to  learn  it? 

This  concept-development  session  provides  an  opportunity  to  explore 
elements  of  the  “SEE”  function  in  relation  to  a  civilian  cultural  situation  and 
allow  for  a  wide  range  of  responses  from  the  participants.  This  concept- 
development  session  is  for  participants  playing  the  role  of  the  UA 
Reconnaissance  Squadron  Commander. 

The  participants  are  presented  with  a  tactical  scenario,  which  includes  a 
detailed  written  account  of  civilian  activities  in  a  town  known  to  harbor  threat 
personnel  on  existing  blacklists.  They  are  also  able  to  view  a  series  of 
photographs  including  groups  of  armed  persons  congregated  near  the  town’s 
central  hall.  They  have  access  to  maps  along  with  graphic  displays  showing 
their  unit  task  organization  and  additional  assets  available.  They  are  told  that 
the  unit’s  task  in  this  situation  is  to  determine  the  nature  of  the  threat  in  the 
town  and  make  a  report  to  the  Battalion  Commander. 

Data  collection  The  structured  interview,  conducted  by  moderator(s),  elicits  responses  aimed 
at  shaping  concepts  and  tactics,  techniques,  and  procedures  (TTP)  related  to 
this  situation,  including  items  such  as  determining  what  types  of  information 
they  needed,  how  that  information  could  best  be  presented,  what  steps  they 
would  take  to  accomplish  the  task,  and  what,  if  any,  additional  assets  might 
have  enhanced  their  performance. 

The  moderators  collect  additional  process  data  as  they  observe  the  participant 
completing  the  concept-development  session.  They  record  actions  taken  by 
the  participant  such  as  what  information  was  reviewed  and  for  how  long, 
whether  or  not  the  mapping  tool  was  used,  etc.,  that  might  be  helpful  in 
conducting  the  interview  or  in  interpreting  the  participants  responses  to  the 
interview. 

Data  analysis  Data  collected  during  the  structured  interview(s)  will  be  analyzed  by  a  panel 
familiar  with  Future  Force  UA  concepts  developed  thus  far  as  presented  in  the 
UA  Operational  and  Organizational  (O&O)  Plan  and  the  operational 
requirements  document  (ORD)  for  Future  Combat  Systems  (FCS). 

Qualitative  analysis  procedures  and  techniques  will  be  used  and,  as  stated 
previously,  the  results  will  be  used  to  modify  the  concept-development  session 
for  additional  presentations.  At  some  point  determined  by  the  panel,  the 
concept-development  session  will  no  longer  be  administered  and  the  concepts, 
TTP,  lessons  learned,  etc.,  from  all  administrations  of  the  concept- 
development  session  will  be  documented. 


General 

description 
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Structured  1 .  The  unit’s  task  in  this  concept-development  session  was  to  determine  the 
interview  nature  of  the  threat  and  report  to  the  Battalion  Commander.  What  steps 

questions  would  you  take  to  complete  this  task?  (Follow-up  and  clarify  as 

necessary). 

2.  What  additional  assets/capabilities,  if  any,  would  you  liked  to  have  had  to 
complete  this  task,  and  how  would  you  have  employed  those  assets? 

3.  Is  there  information  that  you  did  not  have  that  would  have  helped  you 
complete  this  task?  What  information?  How  could  it  best  be  presented? 

4.  Is  this  an  appropriate  task  for  the  unit  indicated?  Should  it  be  a  higher 
level  task?  Lower  level? 

5.  Follow-up  questions  based  upon  observations  of  the  moderators. 


CONCEPT-DEVELOPMENT  SESSION  2  SPECIFICATION 


Title  Design  intuitive  summary  graphics  for  the  Common  Operational  Picture 

(COP) 

General  This  concept-development  session  provides  an  opportunity  for  participants  to 

description  design  and  create  sample  intuitive  summary  graphics  (ISGs)  for  use  on  future 
planning  maps  and  the  COP.  It  focuses  on  ISGs  for  use  at  Bde  level. 
Subsequent  iterations  may  be  conducted  to  design  ISGs  for  Bn  and 
Company/Troop  level. 

An  ISG  is  a  symbol  or  icon  displayed  as  part  of  a  command  and  control 
system.  They  are  simple  to  understand  and  informative,  and  provide  the 
viewer  with  enough  information  to  quickly  grasp  the  nature  of  the  event,  unit, 
activity,  terrain  feature  or  effect  being  depicted.  Examples  of  some  current 
graphics  that  could  be  considered  as  ISGs  include:  military  unit  symbols  and 
graphic  control  measures,  international  traffic/safety  symbols,  and  the  closed 
envelope  representing  email  on  many  computers.  In  the  near  future  it  will  be 
possible  to  clearly  display  multi-color,  composite  symbols  using  computers. 
The  creation  of  a  set  of  well-defined,  easy  to  understand  ISGs  is  required  to 
ensure  the  military  makes  the  best  use  of  available  computer  capabilities  and 
provides  its  personnel  with  one-look-to-understand  graphics. 

This  session  includes  two  methods  for  the  design/creation  of  intuitive 
summary  graphics.  One  consists  of  the  participant  being  presented  with  a 
series  of  spot  reports  in  text.  The  participant  is  asked  to  design  a  graphic  that 
represents  the  information  in  the  reports.  In  the  second  method  the  participant 
is  presented  with  a  series  of  photographs  and/or  drawings  of  military  activities 
and  asked  to  design  a  graphic  that  represents  the  activities.  In  both  methods 
the  participant  has  a  number  of  tools  available  to  help  with  the  design/creation 
of  the  graphics.  These  include  but  are  not  limited  to  drawing  tools  such  as 
Microsoft  PowerPoint®,  a  set  of  graphics  to  start  with  (such  as  boxes,  lines, 
vehicles,  personnel),  a  white  board  and  markers,  and  current  military 
symbology  manuals.  Additionally,  there  is  graphic  designer  on  hand  to  assist 
in  developing  the  proposed  ISGs  in  PowerPoint  or  a  similar  drawing 
application.  This  will  support  the  recording  and  saving  of  the  ISGs  for  future 
use. 


Data  collection  Data  collected  for  this  concept-development  session  consists  primarily  of  the 
ISGs  produced  by  the  participants.  These  are  collected  either  electronically  or 
by  drawing  or  photographing  the  white  board  used  by  participants.  All  ISGs 
produced  are  digitized  for  storage  and  retrieval.  In  addition,  participants  are 
observed  as  they  complete  the  concept-development  session  and  participate  in 
a  structured  interview  conducted  by  the  moderator(s).  The  interview  will 
primarily  elicit  responses  to  determine  the  decision-making  process 
participants  use  to  design  ISGs. 
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Data  analysis 

Data  from  this  session  will  consist  of  graphic  images.  These  will  be  presented 
to  a  panel  familiar  with  Future  Force  concepts  who  will  select  those  that 
clearly  represent  their  underlying  concept. 

Structured 

interview 

questions 

1 .  Your  task  in  this  concept-development  session  was  to  develop  or  design 
intuitive  summary  graphics  that  reflected  the  situation  presented  either  in 
text  or  graphically.  Can  you  describe  the  thought  process  you  went 
through  for  accomplishing  this? 

2.  Was  the  description  of  the  situation,  either  in  text  or  graphics,  complete 
enough  for  you  to  do  your  task?  Explain. 

3.  Follow-up  questions. 

Note 

To  use  this  concept-development  session  as  a  shell  for  developing  other  ISGs, 
it  was  developed  so  that  the  situation  as  presented  in  text  or  graphics  can  be 
easily  changed.  This  allows  a  virtually  unlimited  range  of  situations  to  be 
developed  and  presented  to  participants  resulting  in  the  production  of  an 
unlimited  number  of  ISGs. 
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CONCEPT-DEVELOPMENT  SESSION  3  SPECIFICATION 


Title  What  information  should  a  COP  include?  (Develop  a  COP) 

General  This  concept-development  session  provides  an  opportunity  to  explore 

description  elements  of  the  “SEE”  function  related  to  the  development  of  a  brigade  COP. 
It  is  designed  for  Future  Force  designers  and  developers? 

In  this  concept-development  session  the  participant(s)  are  shown  a 
representation  of  a  COP  at  brigade  level  that  consists  of  multiple  information 
layers  with  intuitive  summary  graphics  representing  operations  graphics,  the 
threat  situation,  terrain  effects,  etc.  The  participants  are  provided  with  a  brief 
tactical  scenario  and  associated  operations  graphics  on  a  two  dimensional 
map.  They  are  then  shown  various  other  graphical  overlays  (threat,  fires, 
obstacles,  etc.),  intuitive  summary  graphics,  reports,  messages,  and  sensor 
feeds  related  to  the  tactical  situation  and  asked  which  of  the  additional  pieces 
of  information  presented  should  be  used  to  create  the  COP.  They  are  also 
asked  to  recommend  how  the  information  should  be  displayed  and  what 
control  the  user  or  viewer  of  the  COP  should  have  in  selecting  what 
information  is  displayed. 

The  COP  represents  a  current  tactical  situation  for  a  brigade  conducting 
offensive  operations  in  a  major  theater  war  (MTW)  against  a  threat 
representing  the  new  Operational  Environment.  The  participant(s)  are  asked 
to  evaluate  the  usefulness  of  each  layer  individually  and  in  multiple 
combinations.  The  participant(s)  are  encouraged  to  request  different 
combinations  of  layers  during  the  concept-development  session.  The  task  is 
to  provide  suggestions  for  improving  the  clarity  of  the  COP. 

This  concept-development  session  is  highly  interactive  between  the 
participant  and  the  moderator(s).  This  is  necessitated  by  two  factors:  the 
moderator  displays  various  preplanned  information  to  the  participant  and  asks 
for  reactions;  and  the  participant  may  wish  to  display  unplanned  information, 
but  will  probably  not  be  familiar  with  the  concept-development  session 
software  in  order  to  create  the  display  himself.  Thus,  during  the  conduct  of 
the  concept-development  session,  the  moderator  not  only  displays  information 
to  the  participant,  he/she  is  also  recording  the  participants’  observations, 
reactions,  and  inputs  on  an  on-going  basis.  The  moderator  uses  a  structured 
data-collection  form  to  capture  data. 
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Data  collection  The  primary  data  collected  from  this  concept-development  session  are  the 
“finished  COPs,”  i.e.,  the  COPs  that  the  participant  determines  as  the  best 
representation  of  the  situation  at  the  brigade  level.  The  COPs  are  captured 
graphically  with  sufficient  annotations  to  provide  clarity  for  any  subsequent 
analyses.  Additional  data  are  obtained  from  the  interactive  data  collection 
that  takes  place  as  the  concept-development  session  proceeds  and  follow-up 
interviews. 


Data  analysis  The  completed  COPs,  along  with  data  collected  during  the  conduct  of  the 

concept-development  session  and  during  subsequent  interviews,  are  analyzed 
by  a  panel  familiar  with  the  Future  Force  UA  concepts.  Qualitative  analysis 
procedures  and  techniques  are  used  and,  as  stated  previously,  the  results  are 
used  to  modify  the  concept-development  session  for  additional  presentations. 
At  some  point  determined  by  the  panel,  the  concept-development  session  will 
no  longer  be  administered  and  the  concepts  and  recommendations  for 
presenting  the  COP  will  be  documented. 


Structured 

interview 

questions 


1 .  Your  task  in  this  concept-development  session  was  to  develop  a  brigade 
level  COP  for  a  Future  Force  operation.  What  was  your  rationale  for 
including  the  information  you  did?  How  well  do  you  think  the  COP  you 
produced  will  work? 

2.  What,  if  any,  important  information  is  missing  from  the  COP? 

3.  Who  should  be  allowed  to  make  changes  to  the  COP?  Explain. 

4.  Is  there  a  single  COP  that  will  serve  all  echelons?  Explain. 

5.  Follow-up  questions  based  upon  observations  of  the  moderators. 
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CONCEPT-DEVELOPMENT  SESSION  4  SPECIFICATION 


Title  What  is  the  process  for  updating  COPs? 

This  concept-development  session  provides  an  opportunity  to  exercise 
portions  of  the  “SEE”  function  in  relation  to  the  updating  of  the  COP  and 
allow  for  a  wide  range  of  responses  from  the  participant(s).  This  concept- 
development  session  explores  the  preferences,  requirements,  and  ideas  of  the 
participant(s)  centered  on  what  they  see  as  the  process  for  adding,  deleting  or 
modifying  information  appearing  on  the  COP. 

The  participant(s)  are  provided  with  a  graphical  representation  of  a  current 
brigade  COP  (e.g.,  the  product  from  concept-development  session  #3)  and 
given  time  to  review  and  understand  the  information  on  the  COP.  The 
participants  are  provided  with  a  series  of  tactical  reports  and  messages  from 
various  units/personnel  and  are  asked  to  decide  which  ones  should  be  added 
to  the  brigade  COP  and  whom  they  think  should  control  the  updates  to  the 
brigade  COP.  The  participant(s)  are  then  asked  to  describe  what  they  see  as  a 
viable  process  for  updating  the  brigade,  battalion,  or  company  COPs  during 
combat  operations. 


Data  collection  Participants  are  provided  with  a  questionnaire  which  they  fill  out  as  they 

complete  the  concept-development  session.  The  questionnaire  prompts  the 
participant  to  decide,  for  each  of  the  reports  and  messages,  which  should  be 
added  to  the  COP  and  by  whom.  Participants  are  also  observed  by 
moderator(s)  as  they  complete  the  concept-development  session,  after  which 
they  will  participate  in  a  structured  interview.  The  interview  elicits  responses 
aimed  at  shaping  concepts  related  to  updating  the  COP. 


Data  analysis  Data  collected  from  the  participant-completed  questionnaires,  the 

observations  and  during  the  structured  interview(s)  is  analyzed  by  a  panel 
familiar  with  Future  Force  UA  concepts  developed  thus  far  as  presented  in  the 
UA  O&O  and  the  ORD  for  FCS.  Qualitative  analysis  procedures  and 
techniques  are  used  and,  as  stated  previously,  the  results  are  used  to  modify 
the  concept-development  session  for  additional  presentations.  At  some  point 
determined  by  the  panel,  the  concept-development  session  will  no  longer  be 
administered  and  the  concepts,  lessons  learned,  etc.  from  all  administrations 
of  the  concept-development  session  will  be  documented. 


General 

description 
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Structured 

interview 

questions 


1 .  Your  task  in  this  concept-development  session  was  to  determine  what 
information  should  be  added  to  a  brigade  COP  and  by  whom.  In  general, 
are  there  some  rules  or  procedures  you  can  think  of  for  deciding  what 
information  gets  added?  Explain. 

2.  In  general,  are  there  some  rules  for  deciding  who  can  modify  the  COP? 
Explain. 

3.  Do  these  rules  and  procedures  apply  to  COPs  at  other  echelons?  Explain. 

4.  Follow-up  questions  based  upon  observations  of  the  moderators. 


CONCEPT-DEVELOPMENT  SESSION  5  SPECIFICATION 


Title  Use  Commander’s  Critical  Information  Requirements  (CCIR)  to 

determine  sensor  requirements 

This  concept-development  session  provides  an  opportunity  to  explore 
elements  of  the  “SEE”  function  related  to  the  development  of  Future  Force 
sensor  requirements  and  usage.  It  allows  for  a  wide  range  of  responses  from 
the  participants.  The  concept-development  session  is  developed  for 
participants  playing  the  role  of  the  UA  Battalion  Commander. 

The  participants  are  provided  with  a  brief  tactical  scenario  at  battalion  level, 
associated  operations  graphics  on  a  two  dimensional  map,  a  set  of  CCIR 
pertaining  to  the  tactical  situation. 


Data  collection  After  participants  work  through  the  scenario,  they  will  participate  in  a 
structured  interview  conducted  by  moderator(s).  The  interview  elicits 
responses  aimed  at  shaping  concepts  and  TTP  related  to  this  situation 
including  items  such  as  determining  what  types  of  sensors  they  needed  and 
how  they  would  deploy  and  use  those  sensors.  Participants  are  also 
encouraged  to  identify  other  (especially  non-existent)  sensor  capabilities  that 
might  help  answer  the  CCIR.  The  moderators  collect  additional  process  data 
as  they  observe  the  participant  completing  the  concept-development  session. 
They  record  actions  taken  by  the  participant  such  as  what  information  was 
reviewed  and  for  how  long,  etc.  that  might  be  helpful  in  conducting  the 
interview  or  in  interpreting  the  participant’s  responses  to  the  interview. 


Data  analysis  Data  collected  during  the  concept-development  session  and  during  the 

structured  interview(s)  are  analyzed  by  a  panel  familiar  with  Future  Force  UA 
concepts  developed  thus  far  as  presented  in  the  UA  O&O  and  the  ORD  for 
FCS.  Qualitative  analysis  procedures  and  techniques  are  used  and,  as  stated 
previously,  the  results  are  used  to  modify  the  concept-development  session  for 
additional  presentations.  At  some  point  determined  by  the  panel,  the  concept- 
development  session  will  no  longer  be  administered  and  the  concepts,  TTP, 
lessons  learned,  etc.  from  all  administrations  of  the  concept-development 
session  will  be  documented. 


1 .  Your  task  in  this  concept-development  session  was  to  examine  the  tactical 
scenario  and  CCIRs  and  to  decide  what  types  of  sensors  would  be 
important  or  useful  in  responding  to  the  CCIRs. 

2.  What  additional  sensor  capabilities  not  included  in  the  list  would  you 
want  to  have?  How  would  you  use  them? 

3.  Does  the  idea  of  tying  sensor  capabilities  to  CCIRs  make  sense?  Explain. 

4.  Follow-up  questions  based  upon  observations  of  the  moderators. 


Structured 

interview 

questions 


General 

description 
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CONCEPT-DEVELOPMENT  SESSION  6  SPECIFICATION 


Title  Assess  an  urban  area,  what  is  the  right  sensor  to  human  mix  for  ISR? 

General  This  concept-development  session  provides  an  opportunity  to  explore 

description  elements  of  the  “SEE”  function  in  relation  to  the  assessment  of  an  Urban 
Area.  The  participants  will  be  playing  the  role  of  the  UA  Combat  Aviation 
Brigade  (CAB)  RSTA  Company  commander. 

Participants  are  presented  with  a  scenario  that  includes  the  requirement  for  the 
unit  to  conduct  the  initial  assessment  (ISR)  activities  of  a  town  known  to 
harbor  threat  personnel  and  equipment.  Participants  have  access  to  maps  and 
a  series  of  photographs  and  sketches  of  the  town.  Additionally,  they  are 
provided  with  graphic  displays  showing  their  available  assets.  They  are  told 
that  the  unit’s  task  in  this  situation  is  to  conduct  an  assessment  of  the  town 
and  make  a  report  to  the  Battalion  Commander.  The  participants  are  asked  to 
develop  a  concept  for  employing  their  assets.  This  session  focuses  on  the 
optimal  mix  of  robotic  sensors  to  human  reconnaissance  assets. 

Data  collection  After  participants  work  through  the  scenario,  they  participate  in  a  structured 
interview  conducted  by  moderator(s).  The  interview  elicits  responses  aimed 
at  shaping  concepts  and  TTP  related  to  this  situation  including  items  such  as 
determining  what  types  of  information  they  needed,  how  that  information 
could  best  be  presented,  what  steps  they  would  take  to  accomplish  the  task, 
and  what,  if  any,  additional  assets  might  have  enhanced  their  performance. 

The  moderators  collect  additional  process  data  as  they  observe  the  participant 
completing  the  concept-development  session.  They  record  actions  taken  by 
the  participant  that  might  be  helpful  in  conducting  the  interview  or  in 
interpreting  the  participant’s  responses  to  the  interview. 

Data  analysis  Data  collected  during  the  structured  interview(s)  are  analyzed  by  a  panel 

familiar  with  Future  Force  UA  concepts  developed  thus  far  as  presented  in  the 
UA  O&O  and  the  ORD  for  FCS.  Qualitative  analysis  procedures  and 
techniques  are  used  and,  as  stated  previously,  the  results  are  used  to  modify 
the  concept-development  session  for  additional  presentations.  At  some  point 
determined  by  the  panel,  the  session  will  no  longer  be  administered  and  the 
concepts,  TTP,  lessons  learned,  etc.  from  all  administrations  of  the  session  are 
documented. 


D-ll 


Structured 

interview 

questions 


1 .  The  unit’s  task  in  this  concept-development  session  was  to  conduct  an 
assessment  of  the  town  and  make  a  report  to  the  Battalion  Commander. 
What  steps  would  you  take  to  complete  this  task?  (Follow-up  and  clarify 
as  necessary.) 

2.  You  were  asked  to  develop  a  concept  for  employing  your  assets  -  both 
human  and  robotic.  Describe  and  explain  your  concept  for  employing 
assets. 

3.  What  do  you  think  is  the  best  mix  of  human  and  robotic  assets  for 
completing  this  task? 

4.  Follow-up  questions  based  upon  observations  of  the  moderators. 
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CONCEPT-DEVELOPMENT  SESSION  7  SPECIFICATION 


Title 

Which  reconnaissance  asset(s)  are  needed  to  conduct  reconnaissance  in 
and  around  caves  and  cave-like  structures? 

General 

description 

This  concept-development  session  provides  an  opportunity  to  exercise 
portions  of  the  “SEE”  function  in  relation  to  a  tactical  situation  that  requires 
the  reconnaissance  of  a  cave  complex.  This  session  allows  for  a  wide  range 
of  responses  from  the  participants  by  exploring  the  anticipated  requirements 
for  conducting  reconnaissance  of  caves  and  cave-like  structures  (sewers) 
during  combat  operations.  It  is  for  participants  playing  the  role  of  the  UA 
Battalion  Commander. 

The  participants  are  provided  with,  a  brief  tactical  scenario  that  includes  the 
requirement  for  their  battalion  to  conduct  reconnaissance  of  suspected  threat 
locations  in  a  mountainous  area  that  includes  a  series  of  small  and  large  caves. 
They  are  also  provided  with  photos  of  the  exterior  and  interior  of  the  cave 
complex.  They  are  then  asked  to  identify  and  prioritize  the  sensor  capabilities 
they  expect  will  provide  the  best  reconnaissance  results.  The  outcome  of  this 
concept-development  session  is  a  list  of  desired  sensor  capabilities  that  the 
participants  believe  are  best  suited  to  use  in  caves  and  cave-like  structures. 

Data  collection 

After  participants  work  through  the  scenario,  they  will  participate  in  a 
structured  interview  conducted  by  moderator(s).  The  interview  will  elicit 
responses  aimed  at  shaping  concepts  and  TTP  related  to  this  situation 
including  items  such  as  determining  what  types  of  information  they  needed, 
how  that  information  could  best  be  obtained,  and  what  steps  they  would  take 
to  accomplish  the  task. 

Data  analysis 

Data  collected  during  the  structured  interview(s)  are  analyzed  by  a  panel 
familiar  with  Future  Force  UA  concepts  developed  thus  far  as  presented  in  the 
UA  O&O  and  the  ORD  for  FCS.  Qualitative  analysis  procedures  and 
techniques  are  used  and,  as  stated  previously,  the  results  are  used  to  modify 
the  session  for  additional  presentations.  At  some  point  determined  by  the 
panel,  the  concept-development  session  will  no  longer  be  administered  and 
the  concepts,  TTP,  lessons  learned,  etc.  from  all  administrations  of  the 
concept-development  session  will  be  documented. 
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Structured 

interview 

questions 


1 .  Your  task  in  this  concept-development  session  was  to  develop  a  list  of 
desired  sensor  capabilities  that  you  think  are  best  suited  to  use  in  caves 
and  cave-like  structures.  What  information  provided  was  important  in 
helping  you  decide  the  sensor  capabilities  required? 

2.  Was  there  additional  information  that  you  would  have  liked  to  have  had? 
How  would  it  have  assisted  you  in  making  your  decisions? 

3.  What  are  the  most  important  sensor  capabilities  you  would  need  for  the 
situation  described? 

4.  Given  that  you  had  all  of  the  sensor  capabilities  you  have  listed,  describe  a 
typical  scenario  for  how  you  would  deploy  and  use  them  in  this  situation. 

5.  Follow-up  questions. 
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CONCEPT-DEVELOPMENT  SESSION  8  SPECIFICATION 


Title 

Fuse  information  from  multiple  sources 

General 

description 

This  concept-development  session  provides  an  opportunity  to  explore 
elements  of  the  “SEE”  function.  The  session  is  for  participants  playing  the 
role  of  a  UA  combat  assessment  (CA)  Battalion  Commander.  It  will  explore 
the  cognitive  processes  associated  with  the  fusion  of  information  from  various 
sources  and  allows  for  a  wide  range  of  responses  from  the  participants. 

The  participants  are  presented  with  2-3  sets  of  information  related  to  a  tactical 
operation  via  a  computer  and  in  a  variety  of  ways  to  include  text,  video,  audio, 
and  graphics.  The  information  consists  of  tactical  reports  from  recent  threat 
activity  that  is  relevant  to  the  participant’s  unit.  The  information  is  unfiltered 
and  in  some  cases  redundant  or  overlapping.  In  other  instances,  the 
information  is  incomplete  when  considered  alone  but  relevant  when  fused 
with  related  information.  The  participants  are  allowed  to  review  the  situation 
and  all  supporting  materials  for  as  long  as  they  want,  after  which  they  develop 
a  situation  report  or  verbal  summary  and  ISG  for  each  information  set. 

Data  collection 

At  the  completion  of  the  concept-development  session,  participants  will 
participate  in  a  structured  interview  conducted  by  moderator(s).  The 
interview  will  elicit  responses  aimed  at  identifying  the  cognitive  processes 
involved  in  fusing  information  from  a  variety  of  sources  into  a  commonly 
understood  tactical  situation. 

Data  analysis 

Data  collected  during  the  structured  interview(s)  are  analyzed  by  a  panel 
familiar  with  Future  Force  UA  concepts  developed  thus  far  as  presented  in  the 
UA  O&O  and  the  ORD  for  FCS.  Qualitative  analysis  procedures  and 
techniques  are  used  and,  as  stated  previously,  the  results  are  used  to  modify 
the  concept-development  session  for  additional  presentations.  At  some  point 
determined  by  the  panel,  the  concept-development  session  will  no  longer  be 
administered  and  the  concepts,  TTP,  lessons  learned,  etc.  from  all 
administrations  of  the  concept-development  session  will  be  documented. 
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Structured 

interview 

questions 


1 .  Your  task  in  this  concept-development  session  was  to  develop  a  picture  of 
the  tactical  situation  based  on  information  from  a  variety  of  sources.  Can 
you  describe  how  you  went  about  doing  this  (that  is,  can  you  describe  the 
steps  you  took  mentally  in  fusing  the  information  from  the  various 
sources)? 

2.  Did  the  ways  in  which  the  information  was  presented  to  you  help  you  with 
the  fusion  process?  Did  they  hinder  you?  Did  they  make  a  difference  one 
way  or  the  other? 

3.  Are  there  different  ways  you  would  prefer  to  have  the  information 
presented  to  you?  Explain. 

4.  What  information  is  critical  to  the  fusion  process? 

5.  Is  there  information  that  is  routinely  presented  that  is  not  important  to  the 
fusion  process?  Explain. 

6.  Are  there  automated  tools  that  you  can  think  of  that  would  assist  you  with 
the  fusion  process?  Explain. 

7.  Follow-up  questions. 
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CONCEPT-DEVELOPMENT  SESSION  9  SPECIFICATION 


Title  Action  -  reaction  -  counteraction  upon  entering  an  immature  AO 

General  This  concept-development  session  provides  an  opportunity  to  explore 

description  elements  of  the  “SEE”  function  in  relation  to  the  use  of  organic  unit  sensors 
for  terrain  reconnaissance  as  an  initial  task  upon  entering  an  immature  AO. 
The  session  is  for  participants  playing  the  role  of  the  UA  CA  Battalion 
Commander. 

One  premise  for  FCS-equipped  units  is  that  they  will  be  required  to  conduct 
terrain  reconnaissance  using  organic  sensors  as  one  of  their  initial  tasks  upon 
entering  an  AO.  The  goal  of  the  reconnaissance  is  to  update  data  on  the 
terrain  in  the  AO  since  many  maps  and  terrain  databases  will  be  outdated  or 
incomplete.  The  question  this  raises  is:  if  it  is  part  of  our  doctrine  to  conduct 
such  reconnaissance  upon  entering  an  AO,  what  will  the  threat’s  reaction  be 
and  what  should  our  counteraction  be?  For  instance,  if  the  threat  knows  we 
always  do  a  sensor-based  terrain  reconnaissance  at  the  outset  of  operations, 
how  will  they  use  this  information?  Will  they  sit  tight  and  monitor  our  sensor 
use  patterns  and  identify  our  tactics  and  procedures?  If  so,  how  will  we 
counter  this  and  maintain  the  advantage  provided  by  our  sensor  capabilities? 

The  participant(s)  are  presented  with  a  short  tactical  scenario  that  has  their 
unit  entering  an  immature  AO  as  the  lead  unit  of  a  UA  brigade.  They  are  also 
provided  with  a  task  organization  and  system  capabilities  matrix  for  the 
battalion.  The  battalion  is  tasked  with  securing  an  entry  point  and  conducting 
a  detailed  terrain  reconnaissance  of  the  brigade  AO  using  its  organic  sensors 
and  selected  brigade  sensors  attached  to  the  battalion.  The  scenario  includes 
information  concerning  the  threat’s  composition,  disposition  and  recent 
activities.  The  threat  consists  of  a  lightly  equipped  but  sophisticated  thinking 
enemy  who  uses  its  forces  to  strike  at  key  government  and  military  targets 
only.  The  threat  also  conducts  extensive  propaganda  and  recruiting  activities, 
both  of  which  have  been  successful. 

The  task  of  the  participants  in  this  concept-development  session  is  to  do  some 
informed  speculating  about  actions  our  forces  and  threat  forces  are  likely  to 
take  in  the  described  situation.  They  will  be  attempting  to  answer  questions 
such  as:  “What  actions  should  we  take  upon  entry  into  the  AO?  If  we  do  this, 
what  do  you  think  the  threat  will  do?  If  the  threat  does  this,  what  should  our 
response  be?” 
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Data  collection  Participant  responses  will  be  captured  by  the  moderator(s)  throughout  the 

concept-development  session,  moderators  will  record  their  responses/actions 
as  a  military  subject  matter  expert  (SME)  leads  them  through  the  thought 
processes  involved  with  completing  it.  In  addition,  they  participate  in  a 
structured  interview  conducted  by  moderator(s)  after  they  have  completed  the 
concept-development  session.  The  interview  elicits  responses  aimed  at 
clarifying  and/or  expanding  the  data  collected  as  the  concept-development 
session  was  completed. 

Data  analysis  Data  collected  during  the  completion  of  the  concept-development  session  and 
from  the  structured  interview(s)  are  analyzed  by  a  panel  familiar  with  Future 
Force  UA  concepts  developed  thus  far  as  presented  in  the  UA  O&O  and  the 
ORD  for  FCS.  Qualitative  analysis  procedures  and  techniques  are  used  and, 
as  stated  previously,  the  results  are  used  to  modify  the  concept-development 
session  for  additional  presentations.  At  some  point  determined  by  the  panel, 
the  concept-development  session  is  no  longer  to  be  administered,  and  the 
concepts,  TTP,  lessons  learned,  etc.,  from  all  administrations  of  the  concept- 
development  session  are  documented. 

Go  through  the  actions  and  responses  taken  by  the  participant  and  ask  any 
follow-up  questions  needed  to  clarify  or  add  to  the  data  you  recorded  during 
the  completion  of  the  concept-development  session. 


Structured 

interview 

questions 
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CONCEPT-DEVELOPMENT  SESSION  10  SPECIFICATION 


Title  Deploy  a  UA  battalion  in  an  AO  to  conduct  reconnaissance  operations 

General  This  concept-development  session  provides  an  opportunity  to  explore 

description  elements  of  the  “SEE”  function  in  relation  to  TTP  associated  with  the 

deployment  of  a  UA  battalion  and  allow  for  a  wide  range  of  responses  from 
the  participants.  The  concept-development  session  is  for  participants  playing 
the  role  of  the  UA  CA  Battalion  Commander. 

The  participants  are  presented  with  a  tactical  scenario  which  requires  them  to 
decide  how  they  will  employ  the  battalion  over  a  very  large  AO  to  conduct 
reconnaissance  operations  in  a  small  scale  contingency  (SSC).  They  have 
access  to  maps  along  with  graphic  displays  showing  the  threat  situation,  the 
battalion’s  assets  and  various  other  information.  They  are  told  that  the  unit’s 
task  in  this  situation  is  to  deploy  rapidly  throughout  the  AO  in  a  manner  that 
supports  conducting  an  area  reconnaissance. 

Participants  are  allowed  to  review  the  situation  and  all  supporting  materials 
for  as  long  as  they  want,  and,  when  ready,  they  are  asked  to  place  their  units 
on  the  map  in  initial  positions  and  to  add  any  graphic  control  measures 
(GCMs)  needed.  They  are  also  asked  to  prepare  short  ‘task  and  purpose’ 
statements  for  each  unit  to  support  their  concept  of  the  operation. 

Data  collection  Once  participants  complete  the  session,  they  participate  in  a  structured 

interview  conducted  by  moderators).  The  interview  elicits  responses  aimed 
at  shaping  concepts  and  TTP  related  to  this  situation  including  items  such  as 
determining  what  types  of  information  they  needed,  how  that  information 
could  best  be  obtained,  and  what  steps  they  would  take  to  accomplish  the  task. 


Data  analysis  Data  collected  during  the  structured  interview(s)  are  analyzed  by  a  panel 

familiar  with  Future  Force  UA  concepts  developed  thus  far  as  presented  in  the 
UA  O&O  and  the  ORD  for  FCS.  Qualitative  analysis  procedures  and 
techniques  are  used  and,  as  stated  previously,  the  results  are  used  to  modify 
the  concept-development  session  for  additional  presentations.  At  some  point 
determined  by  the  panel,  the  concept-development  session  is  longer  to  be 
administered,  and  the  concepts,  TTP,  lessons  learned,  etc.,  from  all 
administrations  of  the  concept-development  session  will  be  documented. 
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Structured 

interview 

questions 


1 .  Your  unit’s  task  in  this  concept-development  session  was  to  deploy 
rapidly  throughout  the  AO  in  a  manner  that  supports  conducting  an  area 
reconnaissance.  Describe  your  rationale  for  your  deployment  plan. 

2.  How  was  your  plan  influenced  by  the  mix  of  live  and  robotic 
reconnaissance  capabilities  you  had  access  to? 

3.  Was  the  information  you  were  provided  sufficient  to  allow  you  to  create 
your  reconnaissance  plan?  Explain. 

4.  Do  you  think  the  reconnaissance  plan  you  produced  adequately  covers  the 
area  involved?  Explain. 

5.  Follow-up  questions. 
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Administration  Guide  for 

Concept-Development  Sessions  for  the  Future  Force 


Introduction 

The  Army,  as  it  incorporates  new  technology  such  as  robotics,  advanced  sensors,  and 
network-centric  Command,  Control,  Communications,  Computers,  Intelligence,  Surveillance  and 
Reconnaissance  (C4ISR)  systems,  is  beginning  a  transformation  process  on  a  wide  scale.  A  key 
to  success  of  this  transformation  is  a  solid  Army  process  for  concept  development  -  a  means  to 
generate,  elaborate,  refine,  describe,  test,  and  validate  new  Future  Force  concepts  relating  to 
doctrine,  tactics,  techniques,  procedures,  unit  and  team  organization,  job  allocation,  training, 
leader  development,  and  other  aspects  of  technology  integration.  One  method  for  concept 
development  is  provided  in  a  tool  for  presenting  concept-development  sessions  (CDS). 

The  notion  of  a  CDS  is  that  of  a  snapshot,  or  a  series  of  snapshots,  or  a  short  presentation 
that  places  a  decision-making,  problem-solving,  or  evaluation  requirement  on  the  participant(s). 

A  CDS,  unlike  a  simulation,  is  not  required  to  respond  dynamically  to  participant  input.  The 
CDS  can  portray  highly  realistic  situations  and  entities  that  are  often  very  difficult  to  model  well 
in  a  simulation,  e.g.,  underground  caves,  car  bombs,  cultural  factors,  and  situations  arising  from 
support  and  stability  operations.  The  CDSs  comprise  the  presentation  of  a  detailed,  realistic 
situation  during  which  a  participant  provides  concepts  and  ideas  related  to  the  situation 
portrayed.  This  may  then  be  followed  by  “what  now?”  or  “what  if?”  queries  to  the  participant(s). 
By  running  a  CDS  multiple  times,  modifying  it  based  on  the  information  collected  from 
participants,  and  repeating  this  run-modify  process,  it  will  be  possible  to  develop  and  fine-tune 
ideas  and  concepts  which  can  then  be  used  as  the  basis  for  additional  testing  or  development 
using  other  tools  that  may  be  developed. 

Structure  of  CDSs 

There  are  currently  10  CDSs  which  have  been  developed  using  RoboHTML  and 
PowerPoint.  They  are  designed  for  execution  on  a  system  that  includes  dual  monitors.  (It  is 
possible  to  present  a  CDS  on  a  single-monitor  configuration;  however,  its  effectiveness  will  be 
seriously  degraded.) 

In  a  typical  CDS,  the  component  developed  in  RoboHTML  is  presented  on  the  left  screen, 
and  the  component  developed  in  PowerPoint  is  presented  on  the  right.  The  RoboHTML 
component  provides  descriptive  information,  often  text-based  although  some  is  presented 
graphically  as  well,  that  sets  the  context  for  the  session  and  provides  the  background  and  other 
details  the  participant  needs  to  start  the  “thought”  process  that  is  at  the  heart  of  the  CDS.  The 
PowerPoint  component  includes  the  graphics  (maps  and  overlays,  pictures,  etc.)  that  support  the 
CDS  and  provide  the  richness  of  detail  that  helps  guide  the  participant’s  thought  process. 

As  is  probably  obvious  by  now,  it  is  the  participants’  “thoughts”  that  are  the  heart  of  the 
CDS.  The  CDS  is  designed  to  capture  these  thoughts  both  during  and  after  the  execution  of  the 
CDS.  This  is  accomplished  in  two  ways.  First,  the  complete  CDS  session  should  be  videotaped 
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(or  at  least  audiotaped)  so  that  it  can  be  analyzed  for  relevant  concepts  and  ideas  that  may  have 
been  elicited.  In  addition,  the  observers/moderators  should  take  notes  during  the  execution  of  the 
CDS,  and  should  interview  the  participant  after  completing  the  CDS.  Each  of  the  CDSs  provided 
includes  a  data  collection  instrument  to  assist  the  observers/moderators  in  collecting  data  during 
the  CDS.  A  list  of  the  ten  CDSs  currently  developed,  along  with  the  echelon  for  which  it  was 
developed  and  its  focus,  is  provided  in  Table  E-l . 

“Running”  a  CDS 

Running  a  CDS  is  a  straightforward  process.  The  participant(s)  should  be  seated  at  the 
display  system;  the  observers/moderators  should  be  seated  in  a  position  where  they  can  easily 
observe  the  actions  being  taken  without  being  intrusive.  There  must  be  at  least  one 
observer/moderator  although  two  are  recommended.  It  is  strongly  recommended  that  at  least  one 
observer/moderator  be  a  military  subject  matter  expert  with  a  good  knowledge  of  Future  Force 
and  FCS  capabilities. 

Prior  to  starting  the  session,  copies  of  the  data  collection  form  for  the  CDS  should  be 
provided  to  each  observer/moderator.  Use  of  this  guide  will  be  discussed  shortly. 

At  the  start  of  a  session,  the  participant(s)  should  be  briefed  on  the  CDS  process.  This 
should  include  a  statement  emphasizing  that  the  CDS  is  not  an  exercise  to  be  solved,  but  rather,  it 
is  a  description  of  a  situation  designed  to  start  the  participant(s)  thinking  about  how  the  Army 
might  function  in  the  future.  The  briefing  should  also  emphasize  that  the  data  the  CDS  is 
designed  to  collect  are  the  “thoughts”  of  the  participant(s).  For  this  reason,  participant(s)  are 
encouraged  to  “think  out  loud”  as  they  go  through  the  CDS.  Some  participant(s)  will  be  more 
comfortable  doing  this  than  others,  and  those  who  prefer  not  to  do  it  should  not  be  forced  to  do 
so.  The  participant(s)  should  be  encouraged  to  think  “future”  and  should  be  told  that  there  are  no 
correct  or  incorrect  answers.  The  purpose  is  to  “see  what  they  think.”  Following  the  initial  brief, 
the  observer/moderator  should  start  the  CDS. 

The  initial  page  or  screen  of  each  CDS  provides  general  instructions  for  completing  it  as 
well  as  describing  the  specific  task  the  participant(s)  is  completing.  After  reading  this  page,  the 
participant(s)  will  start  the  CDS.  As  the  session  proceeds,  the  observer/moderator  should  record 
significant  actions  the  participant(s)  takes  as  well  as  what  he  or  she  is  saying,  questions  he  or  she 
asks,  etc.  using  the  data  collection  form  provided  for  each  CDS.  If  appropriate,  and  this  is  a 
judgment  call,  the  observer/moderator  can  direct  the  participant(s)  to  certain  features  or  aspects 
of  the  CDS  or  can  even  ask  direct  questions  to  attempt  to  lead  the  participant(s)  in  a  specific 
direction.  You  need  to  be  careful  if  you  do  this  however,  since  you  don’t  want  to  unduly 
influence  the  thought  processes  of  the  participant(s)  or  influence  them  to  think  the  way  others 
have  responded  or  the  way  you  think.  A  good  rule  of  thumb  to  remember  is  “you  already  know 
what  you  think;  the  purpose  of  the  CDS  is  to  find  out  what  they  think.” 

Another  important  point  to  keep  in  mind  while  observing  is  to  be  patient.  If  you  do  ask  a 
question  or  otherwise  prompt  the  participant(s),  give  them  plenty  of  time  to  think  about  their 
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response.  Avoid  the  temptation  to  ask  a  quick  follow-up  question  if  they  don’t  respond 
immediately. 

Also  keep  in  mind  that  you  do  not  need  to  ask  every  question  immediately  as  it  occurs  to 
you.  If  a  question  comes  to  mind  that  you  can  ask  after  the  participant(s)  has  completed  the 
CDS,  write  it  down  on  the  observation  form.  Your  purpose  is  to  observe,  not  to  unduly  influence 
-  to  allow  the  participant(s)  to  fully  explore  their  line  of  thought  without  interrupting  in  a  way 
that  takes  them  into  a  different  line  of  thought. 

It  is  anticipated  that  the  CDS  will  go  through  a  progressive  life  cycle.  Specifically,  it  will  be 
administered  several  times  with  single  participants.  The  ideas  gleaned  from  those 
administrations  might  then  be  incorporated  into  the  CDS,  and  it  will  be  administered  with  more 
participants,  possibly  two  or  more  at  a  time.  As  more  and  more  data  are  obtained  the  CDS  is 
modified  and  administered  until  it  reaches  a  point  where  no  new  information  is  forthcoming.  At 
this  point,  the  CDS  has  probably  told  you  everything  it  is  going  to  as  far  as  concept  development 
is  concerned,  and  it  may  well  become  the  basis  for  some  kind  of  training  exercise  or  other  Future 
Force  development  tool. 


Analyzing  Data  from  a  CDS 

The  final  part  of  CDS  administration  involves  analyzing  the  data  obtained  from  the 
participant(s).  Since  each  administration  of  a  CDS  will  probably  be  different  from  all  other 
administrations  of  that  CDS,  it  is  difficult  to  have  a  specific  set  of  items  that  you  can  record  and 
enter  into  a  database.  It  is  much  more  likely  that  you  will  go  back  over  the  responses  produced  in 
an  administration  of  a  CDS  and  “content  analyze”  them  for  important  or  critical  points.  It  is 
these  points  that  would  be  entered  into  a  database  to  allow  you  to  go  back  later  and  look  for 
commonalities  across  participants,  critical  ideas,  etc.  Just  keep  in  mind  that  if  you  change  or 
modify  the  CDS  substantially,  you  will  need  to  be  careful  about  comparing  responses  from  the 
two  different  versions. 
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Table  E-l.  Current  Concept-Development  Sessions. 
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Modifying  a  CDS  and  Developing  a  New  CDS 

It  is  not  difficult  to  modify  a  CDS,  although  it  does  require  knowledge  of  RoboHTML  and 
PowerPoint.  If  you  simply  want  to  change  the  information  on  a  given  page,  you  can  edit  the 
appropriate  file  from  the  list  of  files  that  were  developed  to  make  up  the  compiled  CDS.  More 
extensive  modifications  could  involve  making  new  files  using  RoboHTML  and/or  new 
PowerPoint  slides. 

Developing  a  new  CDS  involves  starting  from  scratch  with  designing  the  CDS  as  the  first 
step.  It  is  important  to  develop  design  specifications  that  include  a  description  of  the  CDS 
scenario,  the  focus  of  the  CDS,  the  concepts  it  is  intended  to  explore,  and  so  forth.  Once  the 
design  is  complete,  development  can  proceed  using  RoboHTML  and  PowerPoint. 

Keep  in  mind  that  if  you  do  significantly  modify  a  CDS  or  create  a  new  one,  you  will  need 
to  develop  appropriate  data  collection  instruments. 
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